[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260812 19:11:08.320734 19738 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.70.190:46041
I20260812 19:11:08.322559 19738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 19:11:08.323854 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 19:11:08.347394 19747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:11:08.347424 19750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:11:08.350406 19738 server_base.cc:1034] running on GCE node
W20260812 19:11:08.349939 19746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:11:09.578809 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:11:09.579102 19738 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:11:09.579324 19738 hybrid_clock.cc:648] HybridClock initialized: now 1786561869579302 us; error 0 us; skew 500 ppm
I20260812 19:11:09.580291 19738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:11:09.588582 19738 webserver.cc:460] Webserver started at http://127.19.70.190:45655/ using document root <none> and password file <none>
I20260812 19:11:09.589885 19738 fs_manager.cc:359] Metadata directory not provided
I20260812 19:11:09.590219 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:11:09.590868 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:11:09.598238 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/master-0-root/instance:
uuid: "4f3e629f8dc34a6c86f47792c7ff5c24"
format_stamp: "Formatted at 2026-08-12 19:11:09 on dist-test-slave-pgkr"
I20260812 19:11:09.608644 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260812 19:11:09.615222 19758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:09.616837 19738 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 19:11:09.617333 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/master-0-root
uuid: "4f3e629f8dc34a6c86f47792c7ff5c24"
format_stamp: "Formatted at 2026-08-12 19:11:09 on dist-test-slave-pgkr"
I20260812 19:11:09.617743 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:11:09.670100 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 19:11:09.672045 19738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 19:11:09.672654 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:11:09.767118 19738 rpc_server.cc:307] RPC server started. Bound to: 127.19.70.190:46041
I20260812 19:11:09.767241 19817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.70.190:46041 every 8 connection(s)
I20260812 19:11:09.776047 19818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:09.804165 19818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24: Bootstrap starting.
I20260812 19:11:09.810935 19818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:09.813369 19818 log.cc:824] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24: Log is configured to *not* fsync() on all Append() calls
I20260812 19:11:09.819487 19818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24: No bootstrap required, opened a new log
I20260812 19:11:09.845467 19818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f3e629f8dc34a6c86f47792c7ff5c24" member_type: VOTER }
I20260812 19:11:09.846168 19818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:09.846385 19818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f3e629f8dc34a6c86f47792c7ff5c24, State: Initialized, Role: FOLLOWER
I20260812 19:11:09.847309 19818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f3e629f8dc34a6c86f47792c7ff5c24" member_type: VOTER }
I20260812 19:11:09.847925 19818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 19:11:09.848197 19818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 19:11:09.848522 19818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:09.853071 19818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f3e629f8dc34a6c86f47792c7ff5c24" member_type: VOTER }
I20260812 19:11:09.853953 19818 leader_election.cc:302] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f3e629f8dc34a6c86f47792c7ff5c24; no voters:
I20260812 19:11:09.856248 19818 leader_election.cc:288] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 19:11:09.856823 19822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:09.860229 19822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [term 1 LEADER]: Becoming Leader. State: Replica: 4f3e629f8dc34a6c86f47792c7ff5c24, State: Running, Role: LEADER
I20260812 19:11:09.861340 19822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f3e629f8dc34a6c86f47792c7ff5c24" member_type: VOTER }
I20260812 19:11:09.862185 19818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:11:09.876101 19824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f3e629f8dc34a6c86f47792c7ff5c24. Latest consensus state: current_term: 1 leader_uuid: "4f3e629f8dc34a6c86f47792c7ff5c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f3e629f8dc34a6c86f47792c7ff5c24" member_type: VOTER } }
I20260812 19:11:09.876633 19823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f3e629f8dc34a6c86f47792c7ff5c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f3e629f8dc34a6c86f47792c7ff5c24" member_type: VOTER } }
I20260812 19:11:09.877218 19824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [sys.catalog]: This master's current role is: LEADER
I20260812 19:11:09.877617 19823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [sys.catalog]: This master's current role is: LEADER
I20260812 19:11:09.883656 19832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 19:11:09.898725 19832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 19:11:09.902227 19738 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260812 19:11:09.922086 19832 catalog_manager.cc:1347] Generated new cluster ID: 970ddb91b26744dea940ebd383d60eed
I20260812 19:11:09.922508 19832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 19:11:09.951751 19832 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 19:11:09.954073 19832 catalog_manager.cc:1504] Loading token signing keys...
I20260812 19:11:09.973564 19832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24: Generated new TSK 0
I20260812 19:11:09.974844 19832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 19:11:10.035073 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 19:11:10.046525 19844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:11:10.047645 19845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:11:10.053411 19738 server_base.cc:1034] running on GCE node
W20260812 19:11:10.053512 19847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:11:10.054590 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:11:10.054889 19738 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:11:10.055106 19738 hybrid_clock.cc:648] HybridClock initialized: now 1786561870055068 us; error 0 us; skew 500 ppm
I20260812 19:11:10.055759 19738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:11:10.067196 19738 webserver.cc:460] Webserver started at http://127.19.70.129:41173/ using document root <none> and password file <none>
I20260812 19:11:10.067875 19738 fs_manager.cc:359] Metadata directory not provided
I20260812 19:11:10.068105 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:11:10.068498 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:11:10.069967 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/ts-0-root/instance:
uuid: "3448dd2b5d4b4a598f259f591c5b3577"
format_stamp: "Formatted at 2026-08-12 19:11:10 on dist-test-slave-pgkr"
I20260812 19:11:10.077225 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.006s
I20260812 19:11:10.082973 19852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:10.084088 19738 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260812 19:11:10.084509 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/ts-0-root
uuid: "3448dd2b5d4b4a598f259f591c5b3577"
format_stamp: "Formatted at 2026-08-12 19:11:10 on dist-test-slave-pgkr"
I20260812 19:11:10.084832 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:11:10.114765 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 19:11:10.116348 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:11:10.119326 19738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 19:11:10.124861 19738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:11:10.125142 19738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:10.125474 19738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 19:11:10.125644 19738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:10.348143 19738 rpc_server.cc:307] RPC server started. Bound to: 127.19.70.129:34063
I20260812 19:11:10.348389 19927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.70.129:34063 every 8 connection(s)
I20260812 19:11:10.358348 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 19:11:10.378917 19933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:11:10.383186 19932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:11:10.387090 19936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:11:10.394713 19738 server_base.cc:1034] running on GCE node
I20260812 19:11:10.396310 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:11:10.396638 19738 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:11:10.396848 19738 hybrid_clock.cc:648] HybridClock initialized: now 1786561870396835 us; error 0 us; skew 500 ppm
I20260812 19:11:10.397723 19738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:11:10.400319 19928 heartbeater.cc:347] Connected to a master server at 127.19.70.190:46041
I20260812 19:11:10.400882 19928 heartbeater.cc:464] Registering TS with master...
I20260812 19:11:10.402197 19738 webserver.cc:460] Webserver started at http://127.19.70.130:43817/ using document root <none> and password file <none>
I20260812 19:11:10.402297 19928 heartbeater.cc:511] Master 127.19.70.190:46041 requested a full tablet report, sending...
I20260812 19:11:10.403179 19738 fs_manager.cc:359] Metadata directory not provided
I20260812 19:11:10.403491 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:11:10.403855 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:11:10.405155 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/ts-1-root/instance:
uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
format_stamp: "Formatted at 2026-08-12 19:11:10 on dist-test-slave-pgkr"
I20260812 19:11:10.407156 19776 ts_manager.cc:194] Registered new tserver with Master: 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063)
I20260812 19:11:10.410949 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 19:11:10.412086 19776 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58608
I20260812 19:11:10.415766 19942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:10.417016 19738 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260812 19:11:10.417368 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/ts-1-root
uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
format_stamp: "Formatted at 2026-08-12 19:11:10 on dist-test-slave-pgkr"
I20260812 19:11:10.417688 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:11:10.442589 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 19:11:10.445199 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:11:10.450320 19738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 19:11:10.459838 19738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:11:10.460201 19738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:10.460477 19738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 19:11:10.460659 19738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:10.725942 19738 rpc_server.cc:307] RPC server started. Bound to: 127.19.70.130:40501
I20260812 19:11:10.726401 20016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.70.130:40501 every 8 connection(s)
I20260812 19:11:10.741952 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 19:11:10.764140 20021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:11:10.766498 20022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:11:10.775378 20025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:11:10.775624 20017 heartbeater.cc:347] Connected to a master server at 127.19.70.190:46041
I20260812 19:11:10.777381 20017 heartbeater.cc:464] Registering TS with master...
I20260812 19:11:10.778878 20017 heartbeater.cc:511] Master 127.19.70.190:46041 requested a full tablet report, sending...
I20260812 19:11:10.780638 19738 server_base.cc:1034] running on GCE node
I20260812 19:11:10.782119 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260812 19:11:10.782106 19776 ts_manager.cc:194] Registered new tserver with Master: 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
W20260812 19:11:10.782545 19738 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:11:10.782836 19738 hybrid_clock.cc:648] HybridClock initialized: now 1786561870782816 us; error 0 us; skew 500 ppm
I20260812 19:11:10.783660 19738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:11:10.784888 19776 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58622
I20260812 19:11:10.787732 19738 webserver.cc:460] Webserver started at http://127.19.70.131:35179/ using document root <none> and password file <none>
I20260812 19:11:10.788410 19738 fs_manager.cc:359] Metadata directory not provided
I20260812 19:11:10.788652 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:11:10.788959 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:11:10.790182 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/ts-2-root/instance:
uuid: "ab6acbd788e445179d8d625d71525849"
format_stamp: "Formatted at 2026-08-12 19:11:10 on dist-test-slave-pgkr"
I20260812 19:11:10.797055 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 19:11:10.801856 20031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:10.803143 19738 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 19:11:10.803563 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/ts-2-root
uuid: "ab6acbd788e445179d8d625d71525849"
format_stamp: "Formatted at 2026-08-12 19:11:10 on dist-test-slave-pgkr"
I20260812 19:11:10.803990 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786561868098184-19738-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:11:10.825941 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 19:11:10.828625 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:11:10.833596 19738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 19:11:10.840556 19738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:11:10.840900 19738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:10.841213 19738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 19:11:10.841447 19738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:11.115440 19738 rpc_server.cc:307] RPC server started. Bound to: 127.19.70.131:43037
I20260812 19:11:11.116097 20105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.70.131:43037 every 8 connection(s)
I20260812 19:11:11.161038 20106 heartbeater.cc:347] Connected to a master server at 127.19.70.190:46041
I20260812 19:11:11.161739 20106 heartbeater.cc:464] Registering TS with master...
I20260812 19:11:11.163053 20106 heartbeater.cc:511] Master 127.19.70.190:46041 requested a full tablet report, sending...
I20260812 19:11:11.166345 19776 ts_manager.cc:194] Registered new tserver with Master: ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:11.166713 19738 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.034006983s
I20260812 19:11:11.169642 19776 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58634
I20260812 19:11:11.176354 19738 test_util.cc:268] Using random seed: 1504928388
I20260812 19:11:11.280491 19776 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58638:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 19:11:11.285346 19776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 19:11:11.420117 19928 heartbeater.cc:503] Master 127.19.70.190:46041 was elected leader, sending a full tablet report...
I20260812 19:11:11.510143 19885 tablet_service.cc:1467] Processing CreateTablet for tablet 97c08be155824b58aa8242e0aa68a62e (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260812 19:11:11.512722 19887 tablet_service.cc:1467] Processing CreateTablet for tablet fb783f78208046f586b7adb1ec2ea95a (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260812 19:11:11.514007 19883 tablet_service.cc:1467] Processing CreateTablet for tablet 1521d1316ec944a5a50b839e70588916 (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260812 19:11:11.509966 19884 tablet_service.cc:1467] Processing CreateTablet for tablet 40f55ca1b3e840adbf76b36e23fae104 (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260812 19:11:11.514667 19886 tablet_service.cc:1467] Processing CreateTablet for tablet da6608b6ce0144edb54e3c4fa602c9cd (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260812 19:11:11.517050 19885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c08be155824b58aa8242e0aa68a62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.530974 19887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb783f78208046f586b7adb1ec2ea95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.528745 19881 tablet_service.cc:1467] Processing CreateTablet for tablet 4e475f91eeb94e0b9bfbd134a1da96fe (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260812 19:11:11.535297 19877 tablet_service.cc:1467] Processing CreateTablet for tablet 90a2c3fe70be421ba0639d24fabe55ca (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260812 19:11:11.540939 19876 tablet_service.cc:1467] Processing CreateTablet for tablet 61c8a97dda724773a4bb69e23875e844 (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260812 19:11:11.539886 19878 tablet_service.cc:1467] Processing CreateTablet for tablet d0e1f458ea8c4061adcdb488b7c02c2f (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260812 19:11:11.550526 19886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6608b6ce0144edb54e3c4fa602c9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.574800 19881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e475f91eeb94e0b9bfbd134a1da96fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.578569 19884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40f55ca1b3e840adbf76b36e23fae104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.576713 19974 tablet_service.cc:1467] Processing CreateTablet for tablet 97c08be155824b58aa8242e0aa68a62e (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260812 19:11:11.567162 19975 tablet_service.cc:1467] Processing CreateTablet for tablet da6608b6ce0144edb54e3c4fa602c9cd (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260812 19:11:11.578919 19973 tablet_service.cc:1467] Processing CreateTablet for tablet 40f55ca1b3e840adbf76b36e23fae104 (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260812 19:11:11.585269 19976 tablet_service.cc:1467] Processing CreateTablet for tablet fb783f78208046f586b7adb1ec2ea95a (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260812 19:11:11.595702 19877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a2c3fe70be421ba0639d24fabe55ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.589965 19971 tablet_service.cc:1467] Processing CreateTablet for tablet 4e475f91eeb94e0b9bfbd134a1da96fe (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260812 19:11:11.612854 19876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c8a97dda724773a4bb69e23875e844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.610522 19974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c08be155824b58aa8242e0aa68a62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.588902 19972 tablet_service.cc:1467] Processing CreateTablet for tablet 1521d1316ec944a5a50b839e70588916 (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260812 19:11:11.603511 20064 tablet_service.cc:1467] Processing CreateTablet for tablet fb783f78208046f586b7adb1ec2ea95a (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260812 19:11:11.616426 19878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e1f458ea8c4061adcdb488b7c02c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.622772 20064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb783f78208046f586b7adb1ec2ea95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.619251 20061 tablet_service.cc:1467] Processing CreateTablet for tablet 40f55ca1b3e840adbf76b36e23fae104 (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260812 19:11:11.619360 19968 tablet_service.cc:1467] Processing CreateTablet for tablet 90a2c3fe70be421ba0639d24fabe55ca (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260812 19:11:11.619556 19969 tablet_service.cc:1467] Processing CreateTablet for tablet d0e1f458ea8c4061adcdb488b7c02c2f (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260812 19:11:11.640203 19883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1521d1316ec944a5a50b839e70588916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.626853 20063 tablet_service.cc:1467] Processing CreateTablet for tablet da6608b6ce0144edb54e3c4fa602c9cd (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260812 19:11:11.634184 19966 tablet_service.cc:1467] Processing CreateTablet for tablet 61c8a97dda724773a4bb69e23875e844 (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260812 19:11:11.640959 20060 tablet_service.cc:1467] Processing CreateTablet for tablet 1521d1316ec944a5a50b839e70588916 (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260812 19:11:11.641593 19973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40f55ca1b3e840adbf76b36e23fae104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.632028 20062 tablet_service.cc:1467] Processing CreateTablet for tablet 97c08be155824b58aa8242e0aa68a62e (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260812 19:11:11.646800 20059 tablet_service.cc:1467] Processing CreateTablet for tablet 4e475f91eeb94e0b9bfbd134a1da96fe (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260812 19:11:11.656038 20058 tablet_service.cc:1467] Processing CreateTablet for tablet d0e1f458ea8c4061adcdb488b7c02c2f (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260812 19:11:11.663156 19968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a2c3fe70be421ba0639d24fabe55ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.664332 20057 tablet_service.cc:1467] Processing CreateTablet for tablet 90a2c3fe70be421ba0639d24fabe55ca (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260812 19:11:11.667871 20061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40f55ca1b3e840adbf76b36e23fae104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.677762 20056 tablet_service.cc:1467] Processing CreateTablet for tablet 61c8a97dda724773a4bb69e23875e844 (DEFAULT_TABLE table=test-workload [id=0f0cce8cc957472fa71fb90dadb0c45d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260812 19:11:11.687336 19972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1521d1316ec944a5a50b839e70588916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.690649 19976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb783f78208046f586b7adb1ec2ea95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.697981 20056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c8a97dda724773a4bb69e23875e844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.703085 19971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e475f91eeb94e0b9bfbd134a1da96fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.708230 20057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a2c3fe70be421ba0639d24fabe55ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.712767 19969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e1f458ea8c4061adcdb488b7c02c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.715646 19966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c8a97dda724773a4bb69e23875e844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.714316 20060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1521d1316ec944a5a50b839e70588916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.743665 19975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6608b6ce0144edb54e3c4fa602c9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.745932 20059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e475f91eeb94e0b9bfbd134a1da96fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.774397 20063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6608b6ce0144edb54e3c4fa602c9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.780669 20062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c08be155824b58aa8242e0aa68a62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.785145 20058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e1f458ea8c4061adcdb488b7c02c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:11.790904 20017 heartbeater.cc:503] Master 127.19.70.190:46041 was elected leader, sending a full tablet report...
W20260812 19:11:11.898368 19833 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260812 19:11:12.006803 20132 tablet_bootstrap.cc:492] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577: Bootstrap starting.
I20260812 19:11:12.035333 20133 tablet_bootstrap.cc:492] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b: Bootstrap starting.
I20260812 19:11:12.070200 20134 tablet_bootstrap.cc:492] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849: Bootstrap starting.
I20260812 19:11:12.072444 20132 tablet_bootstrap.cc:654] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:12.094106 20133 tablet_bootstrap.cc:654] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:12.101100 20134 tablet_bootstrap.cc:654] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:12.103366 20132 tablet_bootstrap.cc:492] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577: No bootstrap required, opened a new log
I20260812 19:11:12.104489 20132 ts_tablet_manager.cc:1397] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577: Time spent bootstrapping tablet: real 0.127s user 0.003s sys 0.040s
I20260812 19:11:12.113734 20133 tablet_bootstrap.cc:492] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b: No bootstrap required, opened a new log
I20260812 19:11:12.113061 20132 raft_consensus.cc:348] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.114705 20133 ts_tablet_manager.cc:1397] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent bootstrapping tablet: real 0.120s user 0.019s sys 0.018s
I20260812 19:11:12.114913 20132 raft_consensus.cc:374] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:12.115978 20132 raft_consensus.cc:729] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Initialized, Role: FOLLOWER
I20260812 19:11:12.117409 20132 consensus_queue.cc:260] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.121383 20134 tablet_bootstrap.cc:492] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849: No bootstrap required, opened a new log
I20260812 19:11:12.122368 20134 ts_tablet_manager.cc:1397] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.034s
I20260812 19:11:12.122341 20133 raft_consensus.cc:348] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.124086 20133 raft_consensus.cc:374] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:12.124708 20133 raft_consensus.cc:729] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Initialized, Role: FOLLOWER
I20260812 19:11:12.126164 20133 consensus_queue.cc:260] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.133155 20134 raft_consensus.cc:348] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } }
I20260812 19:11:12.136500 20134 raft_consensus.cc:374] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:12.136796 20132 ts_tablet_manager.cc:1428] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577: Time spent starting tablet: real 0.031s user 0.012s sys 0.013s
I20260812 19:11:12.137151 20134 raft_consensus.cc:729] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6acbd788e445179d8d625d71525849, State: Initialized, Role: FOLLOWER
I20260812 19:11:12.137555 20133 ts_tablet_manager.cc:1428] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent starting tablet: real 0.022s user 0.015s sys 0.003s
I20260812 19:11:12.139531 20133 tablet_bootstrap.cc:492] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b: Bootstrap starting.
I20260812 19:11:12.138454 20134 consensus_queue.cc:260] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } }
I20260812 19:11:12.140630 20132 tablet_bootstrap.cc:492] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577: Bootstrap starting.
I20260812 19:11:12.159927 20133 tablet_bootstrap.cc:654] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:12.168296 20134 ts_tablet_manager.cc:1428] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849: Time spent starting tablet: real 0.045s user 0.015s sys 0.022s
I20260812 19:11:12.170217 20106 heartbeater.cc:503] Master 127.19.70.190:46041 was elected leader, sending a full tablet report...
I20260812 19:11:12.171020 20134 tablet_bootstrap.cc:492] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849: Bootstrap starting.
I20260812 19:11:12.185434 20132 tablet_bootstrap.cc:654] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:12.203819 20134 tablet_bootstrap.cc:654] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:12.209177 20133 tablet_bootstrap.cc:492] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b: No bootstrap required, opened a new log
I20260812 19:11:12.209853 20133 ts_tablet_manager.cc:1397] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent bootstrapping tablet: real 0.071s user 0.001s sys 0.025s
I20260812 19:11:12.224684 20132 tablet_bootstrap.cc:492] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577: No bootstrap required, opened a new log
I20260812 19:11:12.225551 20132 ts_tablet_manager.cc:1397] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577: Time spent bootstrapping tablet: real 0.087s user 0.015s sys 0.025s
I20260812 19:11:12.227156 20133 raft_consensus.cc:348] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.228753 20133 raft_consensus.cc:374] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:12.229436 20133 raft_consensus.cc:729] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Initialized, Role: FOLLOWER
I20260812 19:11:12.231914 20132 raft_consensus.cc:348] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.232897 20132 raft_consensus.cc:374] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:12.233245 20132 raft_consensus.cc:729] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Initialized, Role: FOLLOWER
I20260812 19:11:12.233397 20133 consensus_queue.cc:260] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.234308 20132 consensus_queue.cc:260] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.239872 20132 ts_tablet_manager.cc:1428] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260812 19:11:12.241559 20132 tablet_bootstrap.cc:492] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577: Bootstrap starting.
I20260812 19:11:12.256726 20132 tablet_bootstrap.cc:654] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577: Neither blocks nor log segments found. Creating new log.
W20260812 19:11:12.256940 20018 tablet.cc:2369] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:11:12.262995 20133 ts_tablet_manager.cc:1428] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent starting tablet: real 0.053s user 0.000s sys 0.018s
I20260812 19:11:12.269560 20134 tablet_bootstrap.cc:492] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849: No bootstrap required, opened a new log
I20260812 19:11:12.270443 20134 ts_tablet_manager.cc:1397] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849: Time spent bootstrapping tablet: real 0.100s user 0.025s sys 0.037s
I20260812 19:11:12.280798 20133 tablet_bootstrap.cc:492] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b: Bootstrap starting.
I20260812 19:11:12.280368 20134 raft_consensus.cc:348] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.283756 20134 raft_consensus.cc:374] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:12.284323 20134 raft_consensus.cc:729] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6acbd788e445179d8d625d71525849, State: Initialized, Role: FOLLOWER
I20260812 19:11:12.285724 20134 consensus_queue.cc:260] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.293467 20132 tablet_bootstrap.cc:492] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577: No bootstrap required, opened a new log
I20260812 19:11:12.300282 20132 ts_tablet_manager.cc:1397] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.041s
I20260812 19:11:12.300624 20133 tablet_bootstrap.cc:654] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:12.302332 20134 ts_tablet_manager.cc:1428] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849: Time spent starting tablet: real 0.031s user 0.003s sys 0.014s
I20260812 19:11:12.306576 20132 raft_consensus.cc:348] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.307833 20132 raft_consensus.cc:374] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:12.308329 20132 raft_consensus.cc:729] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Initialized, Role: FOLLOWER
I20260812 19:11:12.313095 20134 tablet_bootstrap.cc:492] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849: Bootstrap starting.
I20260812 19:11:12.315228 20132 consensus_queue.cc:260] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.325783 20132 ts_tablet_manager.cc:1428] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577: Time spent starting tablet: real 0.025s user 0.005s sys 0.007s
I20260812 19:11:12.328883 20133 tablet_bootstrap.cc:492] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b: No bootstrap required, opened a new log
I20260812 19:11:12.330535 20132 tablet_bootstrap.cc:492] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577: Bootstrap starting.
I20260812 19:11:12.330168 20133 ts_tablet_manager.cc:1397] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent bootstrapping tablet: real 0.050s user 0.021s sys 0.016s
I20260812 19:11:12.336383 20134 tablet_bootstrap.cc:654] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:12.341486 20133 raft_consensus.cc:348] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } }
I20260812 19:11:12.343659 20133 raft_consensus.cc:374] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:12.344900 20133 raft_consensus.cc:729] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Initialized, Role: FOLLOWER
I20260812 19:11:12.346297 20132 tablet_bootstrap.cc:654] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:12.348467 20133 consensus_queue.cc:260] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } }
I20260812 19:11:12.362922 20139 raft_consensus.cc:482] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:12.363756 20139 raft_consensus.cc:504] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.371703 20133 ts_tablet_manager.cc:1428] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent starting tablet: real 0.040s user 0.011s sys 0.005s
I20260812 19:11:12.373178 20134 tablet_bootstrap.cc:492] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849: No bootstrap required, opened a new log
I20260812 19:11:12.373906 20134 ts_tablet_manager.cc:1397] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849: Time spent bootstrapping tablet: real 0.063s user 0.013s sys 0.024s
I20260812 19:11:12.374003 20139 leader_election.cc:288] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:12.397308 20133 tablet_bootstrap.cc:492] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b: Bootstrap starting.
I20260812 19:11:12.402523 20134 raft_consensus.cc:348] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.404639 20139 raft_consensus.cc:482] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:12.405349 20139 raft_consensus.cc:504] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.403687 20132 tablet_bootstrap.cc:492] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577: No bootstrap required, opened a new log
I20260812 19:11:12.410099 20132 ts_tablet_manager.cc:1397] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577: Time spent bootstrapping tablet: real 0.082s user 0.011s sys 0.021s
I20260812 19:11:12.404280 20134 raft_consensus.cc:374] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:12.411159 20139 leader_election.cc:288] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:12.411259 20134 raft_consensus.cc:729] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6acbd788e445179d8d625d71525849, State: Initialized, Role: FOLLOWER
I20260812 19:11:12.429244 20134 consensus_queue.cc:260] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.431888 20132 raft_consensus.cc:348] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.435482 20132 raft_consensus.cc:374] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:12.436203 20132 raft_consensus.cc:729] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Initialized, Role: FOLLOWER
I20260812 19:11:12.437470 20132 consensus_queue.cc:260] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.436350 20134 ts_tablet_manager.cc:1428] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849: Time spent starting tablet: real 0.062s user 0.012s sys 0.004s
I20260812 19:11:12.443966 20140 raft_consensus.cc:482] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:12.448887 20134 tablet_bootstrap.cc:492] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849: Bootstrap starting.
I20260812 19:11:12.447635 20140 raft_consensus.cc:504] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.454648 20141 raft_consensus.cc:482] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:12.455214 20140 leader_election.cc:288] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:12.452678 20133 tablet_bootstrap.cc:654] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:12.455765 20141 raft_consensus.cc:504] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } }
I20260812 19:11:12.463634 20141 leader_election.cc:288] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:12.476707 20141 raft_consensus.cc:482] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:12.499274 20132 ts_tablet_manager.cc:1428] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577: Time spent starting tablet: real 0.089s user 0.011s sys 0.009s
I20260812 19:11:12.501778 20132 tablet_bootstrap.cc:492] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577: Bootstrap starting.
I20260812 19:11:12.489040 20141 raft_consensus.cc:504] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.505198 20141 leader_election.cc:288] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:12.543916 20134 tablet_bootstrap.cc:654] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:12.572189 20140 raft_consensus.cc:482] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:12.580204 20140 raft_consensus.cc:504] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.589210 20156 raft_consensus.cc:482] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:12.590359 20140 leader_election.cc:288] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:12.590535 20156 raft_consensus.cc:504] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.618734 20132 tablet_bootstrap.cc:654] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:12.622735 20139 raft_consensus.cc:482] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:12.630209 20139 raft_consensus.cc:504] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.636410 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" is_pre_election: true
I20260812 19:11:12.651865 20139 leader_election.cc:288] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:12.656208 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1f458ea8c4061adcdb488b7c02c2f" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" is_pre_election: true
I20260812 19:11:12.660569 20140 raft_consensus.cc:482] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:12.651069 20156 leader_election.cc:288] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:12.661643 20140 raft_consensus.cc:504] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } }
W20260812 19:11:12.666546 19853 leader_election.cc:341] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:12.671768 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6608b6ce0144edb54e3c4fa602c9cd" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" is_pre_election: true
I20260812 19:11:12.674443 20133 tablet_bootstrap.cc:492] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b: No bootstrap required, opened a new log
I20260812 19:11:12.676059 20133 ts_tablet_manager.cc:1397] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent bootstrapping tablet: real 0.290s user 0.031s sys 0.031s
W20260812 19:11:12.684280 19853 leader_election.cc:341] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:12.689191 20140 leader_election.cc:288] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
W20260812 19:11:12.690486 19853 leader_election.cc:341] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:12.701340 20075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1f458ea8c4061adcdb488b7c02c2f" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849" is_pre_election: true
I20260812 19:11:12.699950 20133 raft_consensus.cc:348] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.703469 20077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849" is_pre_election: true
I20260812 19:11:12.705933 20133 raft_consensus.cc:374] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:12.706373 20133 raft_consensus.cc:729] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Initialized, Role: FOLLOWER
I20260812 19:11:12.706264 20074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6608b6ce0144edb54e3c4fa602c9cd" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849" is_pre_election: true
I20260812 19:11:12.706535 20077 raft_consensus.cc:2454] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 0.
I20260812 19:11:12.709219 20133 consensus_queue.cc:260] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
W20260812 19:11:12.711391 19857 leader_election.cc:341] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:12.712303 19857 leader_election.cc:302] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577; no voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849
I20260812 19:11:12.714408 20134 tablet_bootstrap.cc:492] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849: No bootstrap required, opened a new log
W20260812 19:11:12.714887 19857 leader_election.cc:341] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:12.715227 20134 ts_tablet_manager.cc:1397] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849: Time spent bootstrapping tablet: real 0.267s user 0.033s sys 0.059s
I20260812 19:11:12.715677 19857 leader_election.cc:302] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577; no voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849
I20260812 19:11:12.717293 20139 raft_consensus.cc:2726] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:11:12.725733 20139 raft_consensus.cc:2726] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:11:12.726498 20134 raft_consensus.cc:348] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.727835 20134 raft_consensus.cc:374] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:12.728482 20133 ts_tablet_manager.cc:1428] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent starting tablet: real 0.043s user 0.013s sys 0.000s
I20260812 19:11:12.728407 20134 raft_consensus.cc:729] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6acbd788e445179d8d625d71525849, State: Initialized, Role: FOLLOWER
I20260812 19:11:12.731544 20134 consensus_queue.cc:260] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.737214 20134 ts_tablet_manager.cc:1428] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849: Time spent starting tablet: real 0.021s user 0.007s sys 0.004s
I20260812 19:11:12.738726 20133 tablet_bootstrap.cc:492] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b: Bootstrap starting.
I20260812 19:11:12.745186 20156 raft_consensus.cc:482] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:12.746444 20134 tablet_bootstrap.cc:492] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849: Bootstrap starting.
I20260812 19:11:12.755337 20132 tablet_bootstrap.cc:492] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577: No bootstrap required, opened a new log
I20260812 19:11:12.761351 20132 ts_tablet_manager.cc:1397] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577: Time spent bootstrapping tablet: real 0.260s user 0.015s sys 0.041s
I20260812 19:11:12.762439 20156 raft_consensus.cc:504] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.771384 20132 raft_consensus.cc:348] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.772323 20132 raft_consensus.cc:374] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:12.772655 20132 raft_consensus.cc:729] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Initialized, Role: FOLLOWER
I20260812 19:11:12.772579 20156 leader_election.cc:288] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:12.776419 20132 consensus_queue.cc:260] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.794981 20140 raft_consensus.cc:482] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:11:12.777261 19857 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:43037, user_credentials={real_user=slave}} blocked reactor thread for 62398.8us
I20260812 19:11:12.822726 20140 raft_consensus.cc:504] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.827674 19898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb783f78208046f586b7adb1ec2ea95a" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577" is_pre_election: true
I20260812 19:11:12.844369 20132 ts_tablet_manager.cc:1428] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577: Time spent starting tablet: real 0.083s user 0.005s sys 0.007s
I20260812 19:11:12.845446 20140 leader_election.cc:288] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:12.842480 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577" is_pre_election: true
I20260812 19:11:12.827905 19899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e475f91eeb94e0b9bfbd134a1da96fe" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577" is_pre_election: true
I20260812 19:11:12.851464 19895 raft_consensus.cc:2454] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 0.
I20260812 19:11:12.851678 20132 tablet_bootstrap.cc:492] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577: Bootstrap starting.
I20260812 19:11:12.847028 20134 tablet_bootstrap.cc:654] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:12.838984 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb783f78208046f586b7adb1ec2ea95a" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577" is_pre_election: true
I20260812 19:11:12.846138 19897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a2c3fe70be421ba0639d24fabe55ca" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577" is_pre_election: true
I20260812 19:11:12.855429 19898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577" is_pre_election: true
I20260812 19:11:12.836841 19857 leader_election.cc:302] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, ab6acbd788e445179d8d625d71525849; no voters: 85cada24ba8d4fd8848da9cb03a8dd4b
W20260812 19:11:12.857905 19946 leader_election.cc:341] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:12.859004 19898 raft_consensus.cc:2454] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 0.
I20260812 19:11:12.865999 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1521d1316ec944a5a50b839e70588916" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577" is_pre_election: true
W20260812 19:11:12.870049 19946 leader_election.cc:341] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063): Illegal state: must be running to vote when last-logged opid is not known
W20260812 19:11:12.874571 20037 leader_election.cc:341] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:12.875873 20037 leader_election.cc:302] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, ab6acbd788e445179d8d625d71525849; no voters:
W20260812 19:11:12.877115 20037 leader_election.cc:341] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:12.877775 20156 raft_consensus.cc:2781] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260812 19:11:12.873441 19946 leader_election.cc:341] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:12.878425 20156 raft_consensus.cc:482] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:12.879133 20156 raft_consensus.cc:3037] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:12.880156 20167 raft_consensus.cc:2781] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:12.883014 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb783f78208046f586b7adb1ec2ea95a" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" is_pre_election: true
I20260812 19:11:12.884084 19988 raft_consensus.cc:2454] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 0.
I20260812 19:11:12.890245 19899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6608b6ce0144edb54e3c4fa602c9cd" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577" is_pre_election: true
I20260812 19:11:12.890875 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" is_pre_election: true
I20260812 19:11:12.891350 19899 raft_consensus.cc:2454] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 0.
I20260812 19:11:12.886246 20167 raft_consensus.cc:482] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:12.893854 20167 raft_consensus.cc:3037] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:12.896088 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1521d1316ec944a5a50b839e70588916" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" is_pre_election: true
I20260812 19:11:12.896584 20037 leader_election.cc:302] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:12.894114 20133 tablet_bootstrap.cc:654] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:12.899763 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6608b6ce0144edb54e3c4fa602c9cd" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" is_pre_election: true
I20260812 19:11:12.898329 20141 raft_consensus.cc:2781] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:12.931155 20141 raft_consensus.cc:482] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:12.931604 20141 raft_consensus.cc:3037] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Advancing to term 1
W20260812 19:11:12.879230 19857 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:43037, user_credentials={real_user=slave}} blocked reactor thread for 49236.6us
I20260812 19:11:12.933583 20139 raft_consensus.cc:482] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:12.934370 20139 raft_consensus.cc:504] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.935281 20156 raft_consensus.cc:504] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.936003 20134 tablet_bootstrap.cc:492] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849: No bootstrap required, opened a new log
I20260812 19:11:12.937018 20134 ts_tablet_manager.cc:1397] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849: Time spent bootstrapping tablet: real 0.199s user 0.035s sys 0.034s
I20260812 19:11:12.903998 19946 leader_election.cc:302] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, 85cada24ba8d4fd8848da9cb03a8dd4b; no voters:
W20260812 19:11:12.902242 19833 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260812 19:11:12.940992 20139 leader_election.cc:288] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:12.942972 19899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:12.943379 20077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849" is_pre_election: true
I20260812 19:11:12.945405 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" is_pre_election: true
W20260812 19:11:12.945943 19857 leader_election.cc:341] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:12.949481 20132 tablet_bootstrap.cc:654] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577: Neither blocks nor log segments found. Creating new log.
W20260812 19:11:12.953869 19853 leader_election.cc:341] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:12.955467 20140 raft_consensus.cc:2781] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:12.956141 20140 raft_consensus.cc:482] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:12.954668 19853 leader_election.cc:302] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577; no voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849
I20260812 19:11:12.957778 20140 raft_consensus.cc:3037] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:12.958694 20139 raft_consensus.cc:2726] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260812 19:11:12.957510 19946 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:34063, user_credentials={real_user=slave}} blocked reactor thread for 53727.8us
I20260812 19:11:12.957219 20156 leader_election.cc:288] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:12.963178 20134 raft_consensus.cc:348] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.964938 20134 raft_consensus.cc:374] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:12.965658 20134 raft_consensus.cc:729] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6acbd788e445179d8d625d71525849, State: Initialized, Role: FOLLOWER
I20260812 19:11:12.963411 20033 leader_election.cc:302] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849; no voters: 3448dd2b5d4b4a598f259f591c5b3577
W20260812 19:11:12.969833 20033 leader_election.cc:341] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:12.967337 20167 raft_consensus.cc:504] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.971999 20156 raft_consensus.cc:2781] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:12.972651 20156 raft_consensus.cc:482] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:12.973073 19899 raft_consensus.cc:2379] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ab6acbd788e445179d8d625d71525849 in current term 1: Already voted for candidate 3448dd2b5d4b4a598f259f591c5b3577 in this term.
I20260812 19:11:12.973327 20156 raft_consensus.cc:3037] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:12.981444 20077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e475f91eeb94e0b9bfbd134a1da96fe" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849" is_pre_election: true
I20260812 19:11:12.983219 20074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a2c3fe70be421ba0639d24fabe55ca" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849" is_pre_election: true
I20260812 19:11:12.983636 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:12.967537 20134 consensus_queue.cc:260] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:12.987421 20074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb783f78208046f586b7adb1ec2ea95a" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849" is_pre_election: true
I20260812 19:11:12.989873 20077 raft_consensus.cc:2454] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 0.
W20260812 19:11:12.992034 19946 leader_election.cc:341] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:12.993198 20167 leader_election.cc:288] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 election: Requested vote from peers 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:12.993045 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:12.993533 20075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849" is_pre_election: true
W20260812 19:11:12.978557 20033 leader_election.cc:341] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:12.992969 19946 leader_election.cc:302] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b; no voters: 3448dd2b5d4b4a598f259f591c5b3577, ab6acbd788e445179d8d625d71525849
W20260812 19:11:12.996025 19853 leader_election.cc:341] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:12.999435 20133 tablet_bootstrap.cc:492] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b: No bootstrap required, opened a new log
I20260812 19:11:13.000336 20133 ts_tablet_manager.cc:1397] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent bootstrapping tablet: real 0.270s user 0.025s sys 0.010s
I20260812 19:11:13.003309 20033 leader_election.cc:302] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab6acbd788e445179d8d625d71525849; no voters: 3448dd2b5d4b4a598f259f591c5b3577, 85cada24ba8d4fd8848da9cb03a8dd4b
I20260812 19:11:13.021859 20140 raft_consensus.cc:504] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.030114 20173 raft_consensus.cc:2726] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:11:13.023991 20134 ts_tablet_manager.cc:1428] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849: Time spent starting tablet: real 0.087s user 0.022s sys 0.015s
I20260812 19:11:13.023806 20141 raft_consensus.cc:504] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
W20260812 19:11:13.027590 20033 leader_election.cc:341] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:13.033208 20132 tablet_bootstrap.cc:492] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577: No bootstrap required, opened a new log
W20260812 19:11:13.035344 19946 leader_election.cc:341] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:13.050699 19899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6608b6ce0144edb54e3c4fa602c9cd" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:13.036146 20140 leader_election.cc:288] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:13.055503 20172 raft_consensus.cc:2726] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:11:13.047040 20134 tablet_bootstrap.cc:492] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849: Bootstrap starting.
I20260812 19:11:13.044588 20133 raft_consensus.cc:348] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.043123 20132 ts_tablet_manager.cc:1397] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577: Time spent bootstrapping tablet: real 0.192s user 0.020s sys 0.029s
I20260812 19:11:13.045827 20141 leader_election.cc:288] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:13.060004 19946 leader_election.cc:302] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849; no voters: 3448dd2b5d4b4a598f259f591c5b3577
W20260812 19:11:13.044881 20033 leader_election.cc:341] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:13.065990 20033 leader_election.cc:302] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab6acbd788e445179d8d625d71525849; no voters: 3448dd2b5d4b4a598f259f591c5b3577, 85cada24ba8d4fd8848da9cb03a8dd4b
I20260812 19:11:13.067235 20172 raft_consensus.cc:2781] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:13.067694 20172 raft_consensus.cc:482] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:13.068275 20172 raft_consensus.cc:3037] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:13.069989 20077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:13.072121 20077 raft_consensus.cc:2379] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3448dd2b5d4b4a598f259f591c5b3577 in current term 1: Already voted for candidate ab6acbd788e445179d8d625d71525849 in this term.
I20260812 19:11:13.075521 20075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:13.034906 20156 raft_consensus.cc:504] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } }
I20260812 19:11:13.078243 20074 raft_consensus.cc:2379] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85cada24ba8d4fd8848da9cb03a8dd4b in current term 1: Already voted for candidate ab6acbd788e445179d8d625d71525849 in this term.
I20260812 19:11:13.082051 20134 tablet_bootstrap.cc:654] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:13.083338 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6608b6ce0144edb54e3c4fa602c9cd" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:13.074458 19898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:13.084939 19898 raft_consensus.cc:3037] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Advancing to term 1
W20260812 19:11:13.084932 19946 leader_election.cc:341] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:13.062328 19899 raft_consensus.cc:3037] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:13.063253 20133 raft_consensus.cc:374] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:13.082171 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb783f78208046f586b7adb1ec2ea95a" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:13.087878 19946 leader_election.cc:302] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b; no voters: 3448dd2b5d4b4a598f259f591c5b3577, ab6acbd788e445179d8d625d71525849
I20260812 19:11:13.090057 20133 raft_consensus.cc:729] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Initialized, Role: FOLLOWER
I20260812 19:11:13.086519 20156 leader_election.cc:288] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:13.090912 19857 leader_election.cc:302] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577; no voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849
I20260812 19:11:13.092197 20140 raft_consensus.cc:3037] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:13.091631 20133 consensus_queue.cc:260] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
W20260812 19:11:13.091832 20037 leader_election.cc:341] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:13.094712 20167 raft_consensus.cc:2726] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260812 19:11:13.096068 19987 raft_consensus.cc:3037] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:13.102604 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb783f78208046f586b7adb1ec2ea95a" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:13.099098 20141 raft_consensus.cc:2726] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260812 19:11:13.104197 20132 raft_consensus.cc:348] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } }
I20260812 19:11:13.111793 20132 raft_consensus.cc:374] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:13.116183 20132 raft_consensus.cc:729] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Initialized, Role: FOLLOWER
I20260812 19:11:13.124431 19898 raft_consensus.cc:2454] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 1.
I20260812 19:11:13.127725 19946 leader_election.cc:302] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, 85cada24ba8d4fd8848da9cb03a8dd4b; no voters: ab6acbd788e445179d8d625d71525849
I20260812 19:11:13.115326 20172 raft_consensus.cc:504] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.133997 20172 leader_election.cc:288] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:13.130303 20175 raft_consensus.cc:2781] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:13.129185 20140 raft_consensus.cc:2726] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:11:13.135146 20134 tablet_bootstrap.cc:492] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849: No bootstrap required, opened a new log
I20260812 19:11:13.135533 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e475f91eeb94e0b9bfbd134a1da96fe" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:13.124966 20132 consensus_queue.cc:260] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } }
I20260812 19:11:13.140036 20134 ts_tablet_manager.cc:1397] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849: Time spent bootstrapping tablet: real 0.097s user 0.016s sys 0.025s
W20260812 19:11:13.141247 19946 leader_election.cc:341] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:13.139199 20175 raft_consensus.cc:686] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 LEADER]: Becoming Leader. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Running, Role: LEADER
I20260812 19:11:13.137388 19987 raft_consensus.cc:2454] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 1.
I20260812 19:11:13.137560 20074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e475f91eeb94e0b9bfbd134a1da96fe" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:13.144629 20167 raft_consensus.cc:482] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:13.143894 20132 ts_tablet_manager.cc:1428] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577: Time spent starting tablet: real 0.080s user 0.006s sys 0.009s
I20260812 19:11:13.145491 20167 raft_consensus.cc:504] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } }
I20260812 19:11:13.150159 20074 raft_consensus.cc:3037] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:13.149992 20134 raft_consensus.cc:348] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.152798 20075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb783f78208046f586b7adb1ec2ea95a" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849" is_pre_election: true
I20260812 19:11:13.153936 20167 leader_election.cc:288] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:13.155102 20075 raft_consensus.cc:2379] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3448dd2b5d4b4a598f259f591c5b3577 in current term 1: Already voted for candidate ab6acbd788e445179d8d625d71525849 in this term.
I20260812 19:11:13.152470 20134 raft_consensus.cc:374] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:13.156078 20134 raft_consensus.cc:729] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6acbd788e445179d8d625d71525849, State: Initialized, Role: FOLLOWER
I20260812 19:11:13.159329 20133 ts_tablet_manager.cc:1428] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent starting tablet: real 0.158s user 0.024s sys 0.029s
I20260812 19:11:13.143277 20175 consensus_queue.cc:237] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.164043 20134 consensus_queue.cc:260] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.166708 20132 tablet_bootstrap.cc:492] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577: Bootstrap starting.
I20260812 19:11:13.165263 19985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb783f78208046f586b7adb1ec2ea95a" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" is_pre_election: true
I20260812 19:11:13.170497 20133 tablet_bootstrap.cc:492] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b: Bootstrap starting.
I20260812 19:11:13.171810 19853 leader_election.cc:302] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577; no voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849
I20260812 19:11:13.180714 20033 leader_election.cc:302] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:13.182812 20141 raft_consensus.cc:2781] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:13.183439 20167 raft_consensus.cc:3037] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:13.183784 20141 raft_consensus.cc:686] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Becoming Leader. State: Replica: ab6acbd788e445179d8d625d71525849, State: Running, Role: LEADER
I20260812 19:11:13.186008 20141 consensus_queue.cc:237] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.193493 19988 raft_consensus.cc:2454] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 1.
I20260812 19:11:13.181320 20134 ts_tablet_manager.cc:1428] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849: Time spent starting tablet: real 0.041s user 0.008s sys 0.003s
I20260812 19:11:13.205957 20134 tablet_bootstrap.cc:492] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849: Bootstrap starting.
I20260812 19:11:13.210744 20033 leader_election.cc:302] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849; no voters: 3448dd2b5d4b4a598f259f591c5b3577
I20260812 19:11:13.246476 20074 raft_consensus.cc:2454] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 1.
I20260812 19:11:13.236780 19899 raft_consensus.cc:2454] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 1.
I20260812 19:11:13.248624 19946 leader_election.cc:302] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849; no voters: 3448dd2b5d4b4a598f259f591c5b3577
I20260812 19:11:13.250520 20140 raft_consensus.cc:2781] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:13.252646 20140 raft_consensus.cc:686] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 LEADER]: Becoming Leader. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Running, Role: LEADER
I20260812 19:11:13.259258 20140 consensus_queue.cc:237] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.232795 20141 raft_consensus.cc:2781] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:13.276625 20141 raft_consensus.cc:686] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Becoming Leader. State: Replica: ab6acbd788e445179d8d625d71525849, State: Running, Role: LEADER
I20260812 19:11:13.282109 20167 raft_consensus.cc:2726] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260812 19:11:13.284904 20033 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:40501, user_credentials={real_user=slave}} blocked reactor thread for 74729.4us
I20260812 19:11:13.284161 20141 consensus_queue.cc:237] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } }
I20260812 19:11:13.295876 19775 catalog_manager.cc:5495] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab6acbd788e445179d8d625d71525849 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "ab6acbd788e445179d8d625d71525849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: HEALTHY } } }
I20260812 19:11:13.310293 20134 tablet_bootstrap.cc:654] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:13.315243 20133 tablet_bootstrap.cc:654] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:13.336092 20132 tablet_bootstrap.cc:654] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:13.357542 20134 tablet_bootstrap.cc:492] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849: No bootstrap required, opened a new log
I20260812 19:11:13.358325 20134 ts_tablet_manager.cc:1397] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849: Time spent bootstrapping tablet: real 0.156s user 0.014s sys 0.024s
I20260812 19:11:13.358562 20133 tablet_bootstrap.cc:492] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b: No bootstrap required, opened a new log
I20260812 19:11:13.359932 20133 ts_tablet_manager.cc:1397] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent bootstrapping tablet: real 0.191s user 0.004s sys 0.033s
I20260812 19:11:13.369446 20132 tablet_bootstrap.cc:492] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577: No bootstrap required, opened a new log
I20260812 19:11:13.370108 20132 ts_tablet_manager.cc:1397] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577: Time spent bootstrapping tablet: real 0.210s user 0.034s sys 0.037s
I20260812 19:11:13.370168 20134 raft_consensus.cc:348] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.369382 20133 raft_consensus.cc:348] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.371688 20134 raft_consensus.cc:374] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:13.372258 20134 raft_consensus.cc:729] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6acbd788e445179d8d625d71525849, State: Initialized, Role: FOLLOWER
I20260812 19:11:13.372182 20133 raft_consensus.cc:374] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:13.373054 20133 raft_consensus.cc:729] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Initialized, Role: FOLLOWER
I20260812 19:11:13.373593 20134 consensus_queue.cc:260] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.376837 20132 raft_consensus.cc:348] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.378754 20134 ts_tablet_manager.cc:1428] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849: Time spent starting tablet: real 0.020s user 0.007s sys 0.006s
I20260812 19:11:13.378852 20132 raft_consensus.cc:374] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:13.379782 20132 raft_consensus.cc:729] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Initialized, Role: FOLLOWER
I20260812 19:11:13.374537 20133 consensus_queue.cc:260] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.380381 20134 tablet_bootstrap.cc:492] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849: Bootstrap starting.
I20260812 19:11:13.381456 20132 consensus_queue.cc:260] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.386344 20132 ts_tablet_manager.cc:1428] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
I20260812 19:11:13.387939 20132 tablet_bootstrap.cc:492] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577: Bootstrap starting.
I20260812 19:11:13.401190 20134 tablet_bootstrap.cc:654] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:13.413050 20133 ts_tablet_manager.cc:1428] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent starting tablet: real 0.052s user 0.007s sys 0.009s
I20260812 19:11:13.414803 20132 tablet_bootstrap.cc:654] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:13.414920 20133 tablet_bootstrap.cc:492] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b: Bootstrap starting.
I20260812 19:11:13.417433 19774 catalog_manager.cc:5495] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130). New cstate: current_term: 1 leader_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:13.430433 20141 raft_consensus.cc:482] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:13.433910 20141 raft_consensus.cc:504] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.439263 19899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a2c3fe70be421ba0639d24fabe55ca" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577" is_pre_election: true
W20260812 19:11:13.442126 20037 leader_election.cc:341] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:13.451920 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a2c3fe70be421ba0639d24fabe55ca" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" is_pre_election: true
I20260812 19:11:13.453327 19988 raft_consensus.cc:2454] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 0.
I20260812 19:11:13.454109 20134 tablet_bootstrap.cc:492] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849: No bootstrap required, opened a new log
I20260812 19:11:13.454867 20134 ts_tablet_manager.cc:1397] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849: Time spent bootstrapping tablet: real 0.075s user 0.030s sys 0.018s
I20260812 19:11:13.446851 19775 catalog_manager.cc:5495] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab6acbd788e445179d8d625d71525849 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "ab6acbd788e445179d8d625d71525849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:13.456132 20033 leader_election.cc:302] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849; no voters: 3448dd2b5d4b4a598f259f591c5b3577
I20260812 19:11:13.457697 20156 raft_consensus.cc:2781] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:13.458333 20156 raft_consensus.cc:482] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:13.458949 20156 raft_consensus.cc:3037] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:13.462954 20141 leader_election.cc:288] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:13.473615 20133 tablet_bootstrap.cc:654] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:13.477707 20132 tablet_bootstrap.cc:492] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577: No bootstrap required, opened a new log
I20260812 19:11:13.477499 20134 raft_consensus.cc:348] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.478462 20134 raft_consensus.cc:374] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:13.478530 20132 ts_tablet_manager.cc:1397] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577: Time spent bootstrapping tablet: real 0.091s user 0.022s sys 0.012s
I20260812 19:11:13.478865 20134 raft_consensus.cc:729] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6acbd788e445179d8d625d71525849, State: Initialized, Role: FOLLOWER
I20260812 19:11:13.479861 20134 consensus_queue.cc:260] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.482223 20134 ts_tablet_manager.cc:1428] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849: Time spent starting tablet: real 0.027s user 0.006s sys 0.006s
I20260812 19:11:13.483192 20134 tablet_bootstrap.cc:492] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849: Bootstrap starting.
I20260812 19:11:13.492381 20156 raft_consensus.cc:504] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.496234 20132 raft_consensus.cc:348] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.497607 20132 raft_consensus.cc:374] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:13.497934 20132 raft_consensus.cc:729] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Initialized, Role: FOLLOWER
I20260812 19:11:13.499080 20134 tablet_bootstrap.cc:654] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:13.500972 19899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a2c3fe70be421ba0639d24fabe55ca" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:13.514569 19774 catalog_manager.cc:5495] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130). New cstate: current_term: 1 leader_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:13.517850 20134 tablet_bootstrap.cc:492] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849: No bootstrap required, opened a new log
I20260812 19:11:13.518759 20134 ts_tablet_manager.cc:1397] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.010s
I20260812 19:11:13.520925 20156 leader_election.cc:288] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:13.521378 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a2c3fe70be421ba0639d24fabe55ca" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:13.522272 19988 raft_consensus.cc:3037] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:13.522294 20132 consensus_queue.cc:260] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.523648 19899 raft_consensus.cc:3037] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:13.536757 20133 tablet_bootstrap.cc:492] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b: No bootstrap required, opened a new log
I20260812 19:11:13.537272 19899 raft_consensus.cc:2454] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 1.
I20260812 19:11:13.538717 19988 raft_consensus.cc:2454] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 1.
I20260812 19:11:13.540484 20033 leader_election.cc:302] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:13.541438 20156 raft_consensus.cc:2781] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:13.542276 20156 raft_consensus.cc:686] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Becoming Leader. State: Replica: ab6acbd788e445179d8d625d71525849, State: Running, Role: LEADER
I20260812 19:11:13.539166 20133 ts_tablet_manager.cc:1397] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent bootstrapping tablet: real 0.125s user 0.019s sys 0.018s
I20260812 19:11:13.541731 20134 raft_consensus.cc:348] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.544119 20134 raft_consensus.cc:374] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:13.543323 20156 consensus_queue.cc:237] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.547592 20134 raft_consensus.cc:729] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6acbd788e445179d8d625d71525849, State: Initialized, Role: FOLLOWER
I20260812 19:11:13.543322 20132 ts_tablet_manager.cc:1428] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577: Time spent starting tablet: real 0.064s user 0.007s sys 0.006s
I20260812 19:11:13.548775 20134 consensus_queue.cc:260] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.549309 20132 tablet_bootstrap.cc:492] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577: Bootstrap starting.
I20260812 19:11:13.557634 20133 raft_consensus.cc:348] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.558804 20133 raft_consensus.cc:374] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:13.559412 20133 raft_consensus.cc:729] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Initialized, Role: FOLLOWER
I20260812 19:11:13.560555 20133 consensus_queue.cc:260] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.562547 20140 raft_consensus.cc:482] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:13.564316 20140 raft_consensus.cc:504] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.566656 20141 raft_consensus.cc:482] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:13.567327 20141 raft_consensus.cc:504] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.569285 20140 leader_election.cc:288] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:13.573025 20074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1521d1316ec944a5a50b839e70588916" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849" is_pre_election: true
I20260812 19:11:13.563560 20134 ts_tablet_manager.cc:1428] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20260812 19:11:13.575734 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577" is_pre_election: true
I20260812 19:11:13.576375 20133 ts_tablet_manager.cc:1428] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent starting tablet: real 0.032s user 0.007s sys 0.003s
I20260812 19:11:13.576896 19895 raft_consensus.cc:2454] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 0.
I20260812 19:11:13.578315 20133 tablet_bootstrap.cc:492] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b: Bootstrap starting.
I20260812 19:11:13.579078 20037 leader_election.cc:302] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:13.575310 20141 leader_election.cc:288] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:13.583174 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" is_pre_election: true
I20260812 19:11:13.584736 20156 raft_consensus.cc:2781] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:13.585283 20156 raft_consensus.cc:482] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:13.585920 20156 raft_consensus.cc:3037] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:13.586155 19899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1521d1316ec944a5a50b839e70588916" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577" is_pre_election: true
I20260812 19:11:13.583913 20132 tablet_bootstrap.cc:654] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:13.587189 19899 raft_consensus.cc:2454] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 0.
W20260812 19:11:13.589879 20033 leader_election.cc:341] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:13.590216 19946 leader_election.cc:302] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, 85cada24ba8d4fd8848da9cb03a8dd4b; no voters:
I20260812 19:11:13.575160 20074 raft_consensus.cc:2454] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 0.
I20260812 19:11:13.591759 20140 raft_consensus.cc:2781] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:13.592429 20140 raft_consensus.cc:482] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:13.593102 20140 raft_consensus.cc:3037] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:13.618950 20156 raft_consensus.cc:504] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.623215 20156 leader_election.cc:288] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:13.623510 20195 raft_consensus.cc:482] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:13.624382 20195 raft_consensus.cc:504] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.625331 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:13.624259 20140 raft_consensus.cc:504] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
W20260812 19:11:13.627815 20033 leader_election.cc:341] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:13.630033 20133 tablet_bootstrap.cc:654] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b: Neither blocks nor log segments found. Creating new log.
W20260812 19:11:13.623234 20037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:34063, user_credentials={real_user=slave}} blocked reactor thread for 44369.4us
I20260812 19:11:13.630748 19899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1521d1316ec944a5a50b839e70588916" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:13.631711 19899 raft_consensus.cc:3037] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:13.635072 20074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1521d1316ec944a5a50b839e70588916" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:13.635558 20175 raft_consensus.cc:482] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:13.636518 20140 leader_election.cc:288] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:13.635993 20074 raft_consensus.cc:3037] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:13.636775 20175 raft_consensus.cc:504] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.642225 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577" is_pre_election: true
I20260812 19:11:13.643524 19895 raft_consensus.cc:2379] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85cada24ba8d4fd8848da9cb03a8dd4b in current term 1: Already voted for candidate 3448dd2b5d4b4a598f259f591c5b3577 in this term.
I20260812 19:11:13.644245 19899 raft_consensus.cc:2454] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 1.
I20260812 19:11:13.645972 19946 leader_election.cc:302] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, 85cada24ba8d4fd8848da9cb03a8dd4b; no voters:
I20260812 19:11:13.647836 20175 leader_election.cc:288] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:13.650894 19899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:13.650710 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577" is_pre_election: true
I20260812 19:11:13.651486 20175 raft_consensus.cc:2781] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:13.652338 19895 raft_consensus.cc:2379] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab6acbd788e445179d8d625d71525849 in current term 1: Already voted for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in this term.
I20260812 19:11:13.652269 20075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849" is_pre_election: true
I20260812 19:11:13.653900 20075 raft_consensus.cc:2379] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85cada24ba8d4fd8848da9cb03a8dd4b in current term 1: Already voted for candidate ab6acbd788e445179d8d625d71525849 in this term.
I20260812 19:11:13.654019 20074 raft_consensus.cc:2454] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 1.
I20260812 19:11:13.652722 20175 raft_consensus.cc:686] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 LEADER]: Becoming Leader. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Running, Role: LEADER
I20260812 19:11:13.654791 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" is_pre_election: true
I20260812 19:11:13.657670 19899 raft_consensus.cc:3037] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:13.658645 19946 leader_election.cc:302] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b; no voters: 3448dd2b5d4b4a598f259f591c5b3577, ab6acbd788e445179d8d625d71525849
I20260812 19:11:13.659811 20033 leader_election.cc:302] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab6acbd788e445179d8d625d71525849; no voters: 3448dd2b5d4b4a598f259f591c5b3577, 85cada24ba8d4fd8848da9cb03a8dd4b
I20260812 19:11:13.660249 20140 raft_consensus.cc:3037] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:13.661370 20156 raft_consensus.cc:3037] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:13.665627 19775 catalog_manager.cc:5495] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab6acbd788e445179d8d625d71525849 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "ab6acbd788e445179d8d625d71525849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: HEALTHY } } }
I20260812 19:11:13.656481 20175 consensus_queue.cc:237] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.649856 20195 leader_election.cc:288] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:13.675238 19899 raft_consensus.cc:2454] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 1.
I20260812 19:11:13.676787 20140 raft_consensus.cc:2726] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:11:13.682024 20132 tablet_bootstrap.cc:492] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577: No bootstrap required, opened a new log
I20260812 19:11:13.699424 20132 ts_tablet_manager.cc:1397] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577: Time spent bootstrapping tablet: real 0.150s user 0.019s sys 0.022s
I20260812 19:11:13.683084 20156 raft_consensus.cc:2726] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:11:13.699741 20037 leader_election.cc:302] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, ab6acbd788e445179d8d625d71525849; no voters: 85cada24ba8d4fd8848da9cb03a8dd4b
I20260812 19:11:13.701257 20156 raft_consensus.cc:2781] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:13.704399 20195 consensus_queue.cc:1035] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:11:13.704823 20132 raft_consensus.cc:348] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.706305 20132 raft_consensus.cc:374] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:13.706739 20132 raft_consensus.cc:729] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Initialized, Role: FOLLOWER
I20260812 19:11:13.707600 20132 consensus_queue.cc:260] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.702876 20156 raft_consensus.cc:686] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Becoming Leader. State: Replica: ab6acbd788e445179d8d625d71525849, State: Running, Role: LEADER
I20260812 19:11:13.716012 20156 consensus_queue.cc:237] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.722954 20132 ts_tablet_manager.cc:1428] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577: Time spent starting tablet: real 0.023s user 0.010s sys 0.000s
I20260812 19:11:13.742597 20195 consensus_queue.cc:1035] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:11:13.752799 20175 consensus_queue.cc:1035] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:11:13.753424 20172 consensus_queue.cc:1035] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:11:13.749395 20133 tablet_bootstrap.cc:492] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b: No bootstrap required, opened a new log
I20260812 19:11:13.757503 20133 ts_tablet_manager.cc:1397] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent bootstrapping tablet: real 0.180s user 0.027s sys 0.032s
I20260812 19:11:13.764369 20133 raft_consensus.cc:348] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.765286 20133 raft_consensus.cc:374] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:13.765692 20133 raft_consensus.cc:729] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Initialized, Role: FOLLOWER
I20260812 19:11:13.765618 20141 consensus_queue.cc:1035] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:11:13.766782 20133 consensus_queue.cc:260] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:13.769964 20133 ts_tablet_manager.cc:1428] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent starting tablet: real 0.012s user 0.005s sys 0.005s
I20260812 19:11:13.771314 20133 tablet_bootstrap.cc:492] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b: Bootstrap starting.
I20260812 19:11:13.769855 19775 catalog_manager.cc:5495] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130). New cstate: current_term: 1 leader_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:13.787942 20172 consensus_queue.cc:1035] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260812 19:11:13.847220 19898 raft_consensus.cc:3037] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:13.912840 19775 catalog_manager.cc:5495] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab6acbd788e445179d8d625d71525849 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "ab6acbd788e445179d8d625d71525849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: HEALTHY } } }
I20260812 19:11:13.926244 20133 tablet_bootstrap.cc:654] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:13.920116 20156 consensus_queue.cc:1035] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260812 19:11:14.004647 20133 tablet_bootstrap.cc:492] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b: No bootstrap required, opened a new log
I20260812 19:11:14.005295 20172 consensus_queue.cc:1035] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:11:14.005596 20212 raft_consensus.cc:482] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:14.007175 20133 ts_tablet_manager.cc:1397] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent bootstrapping tablet: real 0.236s user 0.010s sys 0.039s
I20260812 19:11:14.007802 20212 raft_consensus.cc:504] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:14.016431 20133 raft_consensus.cc:348] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:14.017596 20133 raft_consensus.cc:374] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:14.017974 20133 raft_consensus.cc:729] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Initialized, Role: FOLLOWER
I20260812 19:11:14.019013 20133 consensus_queue.cc:260] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:14.023442 20133 ts_tablet_manager.cc:1428] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b: Time spent starting tablet: real 0.016s user 0.011s sys 0.000s
I20260812 19:11:14.028432 20212 leader_election.cc:288] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:14.031033 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1f458ea8c4061adcdb488b7c02c2f" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" is_pre_election: true
W20260812 19:11:14.032305 19833 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260812 19:11:14.032660 19988 raft_consensus.cc:2454] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 0.
I20260812 19:11:14.030454 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1f458ea8c4061adcdb488b7c02c2f" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577" is_pre_election: true
I20260812 19:11:14.036427 19896 raft_consensus.cc:2454] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 0.
I20260812 19:11:14.037678 20033 leader_election.cc:302] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:14.039170 20212 raft_consensus.cc:2781] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:14.039903 20212 raft_consensus.cc:482] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:14.040791 20212 raft_consensus.cc:3037] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:14.050664 20141 consensus_queue.cc:1035] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:11:14.068140 20212 raft_consensus.cc:504] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:14.072827 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1f458ea8c4061adcdb488b7c02c2f" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:14.073571 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1f458ea8c4061adcdb488b7c02c2f" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:14.073951 19895 raft_consensus.cc:3037] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:14.074512 19987 raft_consensus.cc:3037] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:14.082759 20212 leader_election.cc:288] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:14.083832 19987 raft_consensus.cc:2454] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 1.
I20260812 19:11:14.085394 20033 leader_election.cc:302] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:14.086316 20212 raft_consensus.cc:2781] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:14.086871 20212 raft_consensus.cc:686] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Becoming Leader. State: Replica: ab6acbd788e445179d8d625d71525849, State: Running, Role: LEADER
I20260812 19:11:14.087990 20212 consensus_queue.cc:237] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:14.083832 19895 raft_consensus.cc:2454] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 1.
I20260812 19:11:14.115984 20156 consensus_queue.cc:1035] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:11:14.136931 20172 consensus_queue.cc:1035] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:11:14.158149 19773 catalog_manager.cc:5495] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab6acbd788e445179d8d625d71525849 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "ab6acbd788e445179d8d625d71525849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: HEALTHY } } }
I20260812 19:11:14.173888 20175 consensus_queue.cc:1035] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:11:14.224895 20175 raft_consensus.cc:482] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:14.225592 20175 raft_consensus.cc:504] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:14.228456 20175 leader_election.cc:288] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:14.229494 19898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577" is_pre_election: true
I20260812 19:11:14.229941 20075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849" is_pre_election: true
I20260812 19:11:14.230647 19898 raft_consensus.cc:2379] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85cada24ba8d4fd8848da9cb03a8dd4b in current term 1: Already voted for candidate ab6acbd788e445179d8d625d71525849 in this term.
I20260812 19:11:14.232584 19946 leader_election.cc:302] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b; no voters: 3448dd2b5d4b4a598f259f591c5b3577, ab6acbd788e445179d8d625d71525849
I20260812 19:11:14.233832 20175 raft_consensus.cc:3037] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:14.248404 20175 raft_consensus.cc:2726] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:11:14.253962 20212 consensus_queue.cc:1035] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:11:14.273294 20212 consensus_queue.cc:1035] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:11:14.478770 20212 consensus_queue.cc:1035] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:11:14.494661 20195 consensus_queue.cc:1035] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260812 19:11:15.039279 19833 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260812 19:11:15.051867 20230 raft_consensus.cc:482] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:15.052451 20230 raft_consensus.cc:504] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:15.054404 20230 leader_election.cc:288] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:15.055742 20075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849" is_pre_election: true
I20260812 19:11:15.055809 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" is_pre_election: true
I20260812 19:11:15.056453 20075 raft_consensus.cc:2454] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 1.
I20260812 19:11:15.056565 19988 raft_consensus.cc:2454] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 1.
I20260812 19:11:15.057585 19857 leader_election.cc:302] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:15.058363 20230 raft_consensus.cc:2781] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:11:15.058794 20230 raft_consensus.cc:482] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:15.059077 20230 raft_consensus.cc:3037] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:15.064610 20230 raft_consensus.cc:504] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:15.066290 20230 leader_election.cc:288] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 2 election: Requested vote from peers 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:15.067417 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:15.067423 20075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:15.068254 20075 raft_consensus.cc:3037] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:15.068241 19988 raft_consensus.cc:3037] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:15.073778 19988 raft_consensus.cc:2454] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 2.
I20260812 19:11:15.074095 20075 raft_consensus.cc:2454] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 2.
I20260812 19:11:15.074980 19853 leader_election.cc:302] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, 85cada24ba8d4fd8848da9cb03a8dd4b; no voters:
I20260812 19:11:15.075855 20230 raft_consensus.cc:2781] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:11:15.076877 20230 raft_consensus.cc:686] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Becoming Leader. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Running, Role: LEADER
I20260812 19:11:15.077756 20230 consensus_queue.cc:237] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:15.085995 19773 catalog_manager.cc:5495] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 reported cstate change: term changed from 0 to 2, leader changed from <none> to 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129). New cstate: current_term: 2 leader_uuid: "3448dd2b5d4b4a598f259f591c5b3577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:15.681267 19988 raft_consensus.cc:1261] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Refusing update from remote peer 3448dd2b5d4b4a598f259f591c5b3577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260812 19:11:15.682657 20241 consensus_queue.cc:1035] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:11:15.699004 20075 raft_consensus.cc:1261] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Refusing update from remote peer 3448dd2b5d4b4a598f259f591c5b3577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260812 19:11:15.700973 20241 consensus_queue.cc:1035] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:11:15.730703 20075 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb783f78208046f586b7adb1ec2ea95a"
dest_uuid: "ab6acbd788e445179d8d625d71525849"
mode: GRACEFUL
new_leader_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
from {username='slave'} at 127.0.0.1:53698
I20260812 19:11:15.731429 20075 raft_consensus.cc:595] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Received request to transfer leadership to TS 3448dd2b5d4b4a598f259f591c5b3577
I20260812 19:11:15.737386 20075 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "da6608b6ce0144edb54e3c4fa602c9cd"
dest_uuid: "ab6acbd788e445179d8d625d71525849"
mode: GRACEFUL
new_leader_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
from {username='slave'} at 127.0.0.1:53698
I20260812 19:11:15.737988 20075 raft_consensus.cc:595] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Received request to transfer leadership to TS 3448dd2b5d4b4a598f259f591c5b3577
I20260812 19:11:15.743639 20075 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e"
dest_uuid: "ab6acbd788e445179d8d625d71525849"
mode: GRACEFUL
new_leader_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
from {username='slave'} at 127.0.0.1:53698
I20260812 19:11:15.744259 20075 raft_consensus.cc:595] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Received request to transfer leadership to TS 3448dd2b5d4b4a598f259f591c5b3577
I20260812 19:11:15.760280 19987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104"
dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
mode: GRACEFUL
new_leader_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
from {username='slave'} at 127.0.0.1:53178
I20260812 19:11:15.760823 19987 raft_consensus.cc:595] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 LEADER]: Received request to transfer leadership to TS 3448dd2b5d4b4a598f259f591c5b3577
I20260812 19:11:15.772341 20075 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0e1f458ea8c4061adcdb488b7c02c2f"
dest_uuid: "ab6acbd788e445179d8d625d71525849"
mode: GRACEFUL
new_leader_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
from {username='slave'} at 127.0.0.1:53698
I20260812 19:11:15.772898 20075 raft_consensus.cc:595] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Received request to transfer leadership to TS 85cada24ba8d4fd8848da9cb03a8dd4b
I20260812 19:11:15.778416 20075 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "90a2c3fe70be421ba0639d24fabe55ca"
dest_uuid: "ab6acbd788e445179d8d625d71525849"
mode: GRACEFUL
new_leader_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
from {username='slave'} at 127.0.0.1:53698
I20260812 19:11:15.779055 20075 raft_consensus.cc:595] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Received request to transfer leadership to TS 85cada24ba8d4fd8848da9cb03a8dd4b
I20260812 19:11:15.795977 19898 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844"
dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
mode: GRACEFUL
new_leader_uuid: "ab6acbd788e445179d8d625d71525849"
from {username='slave'} at 127.0.0.1:44250
I20260812 19:11:15.796561 19898 raft_consensus.cc:595] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Received request to transfer leadership to TS ab6acbd788e445179d8d625d71525849
I20260812 19:11:15.916169 20252 raft_consensus.cc:979] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849: : Instructing follower 85cada24ba8d4fd8848da9cb03a8dd4b to start an election
I20260812 19:11:15.916757 20250 raft_consensus.cc:1067] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Signalling peer 85cada24ba8d4fd8848da9cb03a8dd4b to start an election
I20260812 19:11:15.919006 20252 raft_consensus.cc:979] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849: : Instructing follower 85cada24ba8d4fd8848da9cb03a8dd4b to start an election
I20260812 19:11:15.918895 19987 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90a2c3fe70be421ba0639d24fabe55ca"
dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
from {username='slave'} at 127.0.0.1:49236
I20260812 19:11:15.919668 20250 raft_consensus.cc:1067] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Signalling peer 85cada24ba8d4fd8848da9cb03a8dd4b to start an election
I20260812 19:11:15.919730 19987 raft_consensus.cc:482] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:15.920205 19987 raft_consensus.cc:3037] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:15.921543 19988 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0e1f458ea8c4061adcdb488b7c02c2f"
dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
from {username='slave'} at 127.0.0.1:49236
I20260812 19:11:15.922219 19988 raft_consensus.cc:482] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:15.922647 19988 raft_consensus.cc:3037] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:15.927358 19987 raft_consensus.cc:504] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:15.929685 19987 leader_election.cc:288] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 2 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:15.929667 19988 raft_consensus.cc:504] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:15.930660 19898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a2c3fe70be421ba0639d24fabe55ca" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:15.931340 19898 raft_consensus.cc:3037] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:15.931869 19988 leader_election.cc:288] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 2 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:15.932783 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1f458ea8c4061adcdb488b7c02c2f" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:15.933426 19895 raft_consensus.cc:3037] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:15.935272 20075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a2c3fe70be421ba0639d24fabe55ca" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:15.935935 20075 raft_consensus.cc:3032] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:11:15.936143 20074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1f458ea8c4061adcdb488b7c02c2f" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:15.936800 20074 raft_consensus.cc:3032] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:11:15.936311 20075 raft_consensus.cc:729] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab6acbd788e445179d8d625d71525849, State: Running, Role: LEADER
I20260812 19:11:15.937140 20074 raft_consensus.cc:729] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab6acbd788e445179d8d625d71525849, State: Running, Role: LEADER
I20260812 19:11:15.937872 20075 consensus_queue.cc:260] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:15.937901 20074 consensus_queue.cc:260] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:15.938870 19898 raft_consensus.cc:2454] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 2.
I20260812 19:11:15.939443 20074 raft_consensus.cc:3037] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:15.939642 20075 raft_consensus.cc:3037] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:15.941741 19895 raft_consensus.cc:2454] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 2.
I20260812 19:11:15.943104 19946 leader_election.cc:302] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, 85cada24ba8d4fd8848da9cb03a8dd4b; no voters:
I20260812 19:11:15.943979 20199 raft_consensus.cc:2781] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:11:15.944155 19946 leader_election.cc:302] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, 85cada24ba8d4fd8848da9cb03a8dd4b; no voters:
I20260812 19:11:15.947309 20074 raft_consensus.cc:2454] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 2.
I20260812 19:11:15.948691 20075 raft_consensus.cc:2454] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 2.
I20260812 19:11:15.950103 20258 raft_consensus.cc:2781] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:11:15.950681 20258 raft_consensus.cc:686] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 LEADER]: Becoming Leader. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Running, Role: LEADER
I20260812 19:11:15.951607 20258 consensus_queue.cc:237] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:15.953416 20199 raft_consensus.cc:686] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 LEADER]: Becoming Leader. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Running, Role: LEADER
I20260812 19:11:15.954450 20199 consensus_queue.cc:237] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:15.955601 20257 raft_consensus.cc:979] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b: : Instructing follower 3448dd2b5d4b4a598f259f591c5b3577 to start an election
I20260812 19:11:15.956248 20259 raft_consensus.cc:1067] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 LEADER]: Signalling peer 3448dd2b5d4b4a598f259f591c5b3577 to start an election
I20260812 19:11:15.959967 19895 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104"
dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
from {username='slave'} at 127.0.0.1:58164
I20260812 19:11:15.960623 19895 raft_consensus.cc:482] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:15.960912 19895 raft_consensus.cc:3037] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:15.966995 19895 raft_consensus.cc:504] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:15.968878 19774 catalog_manager.cc:5495] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b reported cstate change: term changed from 1 to 2, leader changed from ab6acbd788e445179d8d625d71525849 (127.19.70.131) to 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130). New cstate: current_term: 2 leader_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:15.969676 19895 leader_election.cc:288] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 2 election: Requested vote from peers 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:15.970166 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:15.970890 19988 raft_consensus.cc:3032] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:11:15.971266 19988 raft_consensus.cc:729] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Running, Role: LEADER
I20260812 19:11:15.971038 19774 catalog_manager.cc:5495] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b reported cstate change: term changed from 1 to 2, leader changed from ab6acbd788e445179d8d625d71525849 (127.19.70.131) to 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130). New cstate: current_term: 2 leader_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:15.970381 20075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:15.972090 19988 consensus_queue.cc:260] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:15.972702 20075 raft_consensus.cc:3037] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:15.973501 19988 raft_consensus.cc:3037] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:15.980858 20075 raft_consensus.cc:2454] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 2.
I20260812 19:11:15.982529 19857 leader_election.cc:302] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:15.983626 20241 raft_consensus.cc:2781] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:11:15.984143 20241 raft_consensus.cc:686] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Becoming Leader. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Running, Role: LEADER
I20260812 19:11:15.984822 19988 raft_consensus.cc:2454] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 2.
I20260812 19:11:15.985033 20241 consensus_queue.cc:237] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:15.993870 19774 catalog_manager.cc:5495] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 reported cstate change: term changed from 1 to 2, leader changed from 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130) to 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129). New cstate: current_term: 2 leader_uuid: "3448dd2b5d4b4a598f259f591c5b3577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:16.069873 20252 raft_consensus.cc:979] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849: : Instructing follower 3448dd2b5d4b4a598f259f591c5b3577 to start an election
I20260812 19:11:16.070370 20250 raft_consensus.cc:1067] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Signalling peer 3448dd2b5d4b4a598f259f591c5b3577 to start an election
I20260812 19:11:16.072253 19895 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb783f78208046f586b7adb1ec2ea95a"
dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
from {username='slave'} at 127.0.0.1:58168
I20260812 19:11:16.072903 19895 raft_consensus.cc:482] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:16.073253 19895 raft_consensus.cc:3037] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:16.078924 19895 raft_consensus.cc:504] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } }
I20260812 19:11:16.080722 19895 leader_election.cc:288] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 2 election: Requested vote from peers ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:16.081784 20075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb783f78208046f586b7adb1ec2ea95a" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:16.081881 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb783f78208046f586b7adb1ec2ea95a" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:16.082667 20075 raft_consensus.cc:3032] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:11:16.082855 19988 raft_consensus.cc:3037] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:16.083045 20075 raft_consensus.cc:729] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab6acbd788e445179d8d625d71525849, State: Running, Role: LEADER
I20260812 19:11:16.084045 20075 consensus_queue.cc:260] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } }
I20260812 19:11:16.085203 20075 raft_consensus.cc:3037] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:16.088565 19988 raft_consensus.cc:2454] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 2.
I20260812 19:11:16.089941 19853 leader_election.cc:302] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, 85cada24ba8d4fd8848da9cb03a8dd4b; no voters:
I20260812 19:11:16.090870 20241 raft_consensus.cc:2781] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:11:16.090998 20075 raft_consensus.cc:2454] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 2.
I20260812 19:11:16.091635 20241 raft_consensus.cc:686] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Becoming Leader. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Running, Role: LEADER
I20260812 19:11:16.092625 20241 consensus_queue.cc:237] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } }
I20260812 19:11:16.100062 19774 catalog_manager.cc:5495] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 reported cstate change: term changed from 1 to 2, leader changed from ab6acbd788e445179d8d625d71525849 (127.19.70.131) to 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129). New cstate: current_term: 2 leader_uuid: "3448dd2b5d4b4a598f259f591c5b3577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:16.126399 20250 raft_consensus.cc:979] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849: : Instructing follower 3448dd2b5d4b4a598f259f591c5b3577 to start an election
I20260812 19:11:16.126940 20252 raft_consensus.cc:1067] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Signalling peer 3448dd2b5d4b4a598f259f591c5b3577 to start an election
I20260812 19:11:16.128566 19895 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da6608b6ce0144edb54e3c4fa602c9cd"
dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
from {username='slave'} at 127.0.0.1:58168
I20260812 19:11:16.128834 20250 raft_consensus.cc:979] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849: : Instructing follower 3448dd2b5d4b4a598f259f591c5b3577 to start an election
I20260812 19:11:16.129127 19895 raft_consensus.cc:482] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:16.129316 20252 raft_consensus.cc:1067] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Signalling peer 3448dd2b5d4b4a598f259f591c5b3577 to start an election
I20260812 19:11:16.129482 19895 raft_consensus.cc:3037] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:16.131101 19898 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e"
dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
from {username='slave'} at 127.0.0.1:58168
I20260812 19:11:16.131636 19898 raft_consensus.cc:482] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:16.131934 19898 raft_consensus.cc:3037] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:16.134989 19895 raft_consensus.cc:504] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:16.136904 19895 leader_election.cc:288] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 2 election: Requested vote from peers 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:16.137024 19898 raft_consensus.cc:504] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:16.137853 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6608b6ce0144edb54e3c4fa602c9cd" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:16.138037 20075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6608b6ce0144edb54e3c4fa602c9cd" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:16.138497 19988 raft_consensus.cc:3037] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:16.138525 20075 raft_consensus.cc:3032] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:11:16.139055 20075 raft_consensus.cc:729] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab6acbd788e445179d8d625d71525849, State: Running, Role: LEADER
I20260812 19:11:16.140038 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:16.139886 20075 consensus_queue.cc:260] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:16.140690 19987 raft_consensus.cc:3037] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:16.141072 20075 raft_consensus.cc:3037] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:16.141585 20077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:16.142238 20077 raft_consensus.cc:3032] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:11:16.142583 20077 raft_consensus.cc:729] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab6acbd788e445179d8d625d71525849, State: Running, Role: LEADER
I20260812 19:11:16.143421 20077 consensus_queue.cc:260] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:16.144578 20077 raft_consensus.cc:3037] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:16.148250 19987 raft_consensus.cc:2454] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 2.
I20260812 19:11:16.148538 20075 raft_consensus.cc:2454] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 2.
I20260812 19:11:16.148885 19988 raft_consensus.cc:2454] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 2.
I20260812 19:11:16.149746 19853 leader_election.cc:302] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, 85cada24ba8d4fd8848da9cb03a8dd4b; no voters:
I20260812 19:11:16.149804 19857 leader_election.cc:302] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:16.150441 19898 leader_election.cc:288] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 2 election: Requested vote from peers 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:16.150868 20245 raft_consensus.cc:2781] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:11:16.151342 20245 raft_consensus.cc:686] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Becoming Leader. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Running, Role: LEADER
I20260812 19:11:16.151865 20241 raft_consensus.cc:2781] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:11:16.152115 20245 consensus_queue.cc:237] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:16.153120 20241 raft_consensus.cc:686] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Becoming Leader. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Running, Role: LEADER
I20260812 19:11:16.154026 20241 consensus_queue.cc:237] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:16.157073 20077 raft_consensus.cc:2454] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 2.
I20260812 19:11:16.165256 19774 catalog_manager.cc:5495] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 reported cstate change: term changed from 1 to 2, leader changed from ab6acbd788e445179d8d625d71525849 (127.19.70.131) to 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129). New cstate: current_term: 2 leader_uuid: "3448dd2b5d4b4a598f259f591c5b3577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:16.166544 19774 catalog_manager.cc:5495] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 reported cstate change: term changed from 1 to 2, leader changed from ab6acbd788e445179d8d625d71525849 (127.19.70.131) to 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129). New cstate: current_term: 2 leader_uuid: "3448dd2b5d4b4a598f259f591c5b3577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:16.236668 20245 raft_consensus.cc:979] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577: : Instructing follower ab6acbd788e445179d8d625d71525849 to start an election
I20260812 19:11:16.237224 20243 raft_consensus.cc:1067] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Signalling peer ab6acbd788e445179d8d625d71525849 to start an election
I20260812 19:11:16.238837 20077 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844"
dest_uuid: "ab6acbd788e445179d8d625d71525849"
from {username='slave'} at 127.0.0.1:33876
I20260812 19:11:16.239460 20077 raft_consensus.cc:482] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:16.239881 20077 raft_consensus.cc:3037] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:16.245078 20077 raft_consensus.cc:504] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:16.246785 20077 leader_election.cc:288] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 3 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:16.247916 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:16.248126 19898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c8a97dda724773a4bb69e23875e844" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:16.248595 19987 raft_consensus.cc:3037] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:16.248718 19898 raft_consensus.cc:3032] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:11:16.249069 19898 raft_consensus.cc:729] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Running, Role: LEADER
I20260812 19:11:16.249811 19898 consensus_queue.cc:260] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:16.250928 19898 raft_consensus.cc:3037] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:16.253738 19987 raft_consensus.cc:2454] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 3.
I20260812 19:11:16.254941 20033 leader_election.cc:302] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:16.255831 20252 raft_consensus.cc:2781] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:11:16.256315 20252 raft_consensus.cc:686] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 3 LEADER]: Becoming Leader. State: Replica: ab6acbd788e445179d8d625d71525849, State: Running, Role: LEADER
I20260812 19:11:16.256349 19898 raft_consensus.cc:2454] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 3.
I20260812 19:11:16.257257 20252 consensus_queue.cc:237] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:16.264789 19774 catalog_manager.cc:5495] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 reported cstate change: term changed from 2 to 3, leader changed from 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129) to ab6acbd788e445179d8d625d71525849 (127.19.70.131). New cstate: current_term: 3 leader_uuid: "ab6acbd788e445179d8d625d71525849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: HEALTHY } } }
I20260812 19:11:16.351779 19898 raft_consensus.cc:1261] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Refusing update from remote peer 85cada24ba8d4fd8848da9cb03a8dd4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:16.353209 20259 consensus_queue.cc:1035] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:11:16.362685 20077 raft_consensus.cc:1261] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Refusing update from remote peer 85cada24ba8d4fd8848da9cb03a8dd4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:16.364830 20259 consensus_queue.cc:1035] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:11:16.422386 20077 raft_consensus.cc:1261] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Refusing update from remote peer 85cada24ba8d4fd8848da9cb03a8dd4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:16.424237 20259 consensus_queue.cc:1035] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:16.425503 19987 raft_consensus.cc:1261] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Refusing update from remote peer 3448dd2b5d4b4a598f259f591c5b3577: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:16.427114 20243 consensus_queue.cc:1035] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:11:16.437132 19898 raft_consensus.cc:1261] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Refusing update from remote peer 85cada24ba8d4fd8848da9cb03a8dd4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:16.442430 20259 consensus_queue.cc:1035] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:16.443317 20077 raft_consensus.cc:1261] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Refusing update from remote peer 3448dd2b5d4b4a598f259f591c5b3577: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:16.445011 20241 consensus_queue.cc:1035] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:11:16.540856 20077 raft_consensus.cc:1261] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Refusing update from remote peer 3448dd2b5d4b4a598f259f591c5b3577: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:16.542946 20241 consensus_queue.cc:1035] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:11:16.562347 19988 raft_consensus.cc:1261] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Refusing update from remote peer 3448dd2b5d4b4a598f259f591c5b3577: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:16.565085 20075 raft_consensus.cc:1261] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Refusing update from remote peer 3448dd2b5d4b4a598f259f591c5b3577: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:16.566254 20241 consensus_queue.cc:1035] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:16.568451 20269 consensus_queue.cc:1035] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:16.594808 19988 raft_consensus.cc:1261] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Refusing update from remote peer 3448dd2b5d4b4a598f259f591c5b3577: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:16.597414 20243 consensus_queue.cc:1035] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:16.639827 19988 raft_consensus.cc:1261] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Refusing update from remote peer 3448dd2b5d4b4a598f259f591c5b3577: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:16.641760 20245 consensus_queue.cc:1035] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:16.661706 20075 raft_consensus.cc:1261] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Refusing update from remote peer 3448dd2b5d4b4a598f259f591c5b3577: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:16.664584 20243 consensus_queue.cc:1035] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:16.809237 19988 raft_consensus.cc:1261] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Refusing update from remote peer ab6acbd788e445179d8d625d71525849: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260812 19:11:16.810750 20252 consensus_queue.cc:1035] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:11:16.821915 19898 raft_consensus.cc:1261] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Refusing update from remote peer ab6acbd788e445179d8d625d71525849: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260812 19:11:16.823603 20252 consensus_queue.cc:1035] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:18.832161 19738 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260812 19:11:18.833515 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:18.855595 19988 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1521d1316ec944a5a50b839e70588916"
dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
mode: GRACEFUL
new_leader_uuid: "ab6acbd788e445179d8d625d71525849"
from {username='slave'} at 127.0.0.1:53186
I20260812 19:11:18.856411 19988 raft_consensus.cc:595] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 LEADER]: Received request to transfer leadership to TS ab6acbd788e445179d8d625d71525849
I20260812 19:11:18.872027 19898 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104"
dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
mode: GRACEFUL
new_leader_uuid: "ab6acbd788e445179d8d625d71525849"
from {username='slave'} at 127.0.0.1:44260
I20260812 19:11:18.872679 19898 raft_consensus.cc:595] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Received request to transfer leadership to TS ab6acbd788e445179d8d625d71525849
I20260812 19:11:18.874068 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260812 19:11:18.874559 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:19.158951 20259 raft_consensus.cc:979] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b: : Instructing follower ab6acbd788e445179d8d625d71525849 to start an election
I20260812 19:11:19.159600 20297 raft_consensus.cc:1067] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 LEADER]: Signalling peer ab6acbd788e445179d8d625d71525849 to start an election
I20260812 19:11:19.161515 20075 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1521d1316ec944a5a50b839e70588916"
dest_uuid: "ab6acbd788e445179d8d625d71525849"
from {username='slave'} at 127.0.0.1:33884
I20260812 19:11:19.162089 20075 raft_consensus.cc:482] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:19.162460 20075 raft_consensus.cc:3037] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:19.169816 20075 raft_consensus.cc:504] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:19.171725 20075 leader_election.cc:288] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 2 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:19.172868 19898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1521d1316ec944a5a50b839e70588916" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:19.172966 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1521d1316ec944a5a50b839e70588916" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:19.173671 19898 raft_consensus.cc:3037] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:19.173804 19988 raft_consensus.cc:3032] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:11:19.174221 19988 raft_consensus.cc:729] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Running, Role: LEADER
I20260812 19:11:19.175060 19988 consensus_queue.cc:260] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:19.176244 19988 raft_consensus.cc:3037] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:19.180033 19898 raft_consensus.cc:2454] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 2.
I20260812 19:11:19.181672 19988 raft_consensus.cc:2454] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 2.
I20260812 19:11:19.181569 20037 leader_election.cc:302] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:19.182947 20252 raft_consensus.cc:2781] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:11:19.184238 20252 raft_consensus.cc:686] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 2 LEADER]: Becoming Leader. State: Replica: ab6acbd788e445179d8d625d71525849, State: Running, Role: LEADER
I20260812 19:11:19.185443 20252 consensus_queue.cc:237] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:19.194336 19774 catalog_manager.cc:5495] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 reported cstate change: term changed from 1 to 2, leader changed from 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130) to ab6acbd788e445179d8d625d71525849 (127.19.70.131). New cstate: current_term: 2 leader_uuid: "ab6acbd788e445179d8d625d71525849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: HEALTHY } } }
I20260812 19:11:19.256968 20245 raft_consensus.cc:979] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577: : Instructing follower ab6acbd788e445179d8d625d71525849 to start an election
I20260812 19:11:19.257545 20304 raft_consensus.cc:1067] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Signalling peer ab6acbd788e445179d8d625d71525849 to start an election
I20260812 19:11:19.259271 20075 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104"
dest_uuid: "ab6acbd788e445179d8d625d71525849"
from {username='slave'} at 127.0.0.1:33876
I20260812 19:11:19.259814 20075 raft_consensus.cc:482] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:19.260144 20075 raft_consensus.cc:3037] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:19.265362 20075 raft_consensus.cc:504] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:19.267266 20075 leader_election.cc:288] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 3 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:19.268217 19898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:19.268460 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:19.268818 19898 raft_consensus.cc:3032] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:11:19.269008 19988 raft_consensus.cc:3037] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:19.269181 19898 raft_consensus.cc:729] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Running, Role: LEADER
I20260812 19:11:19.269990 19898 consensus_queue.cc:260] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:19.271090 19898 raft_consensus.cc:3037] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:19.273521 19988 raft_consensus.cc:2454] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 3.
I20260812 19:11:19.274753 20033 leader_election.cc:302] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:19.275547 20252 raft_consensus.cc:2781] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:11:19.276011 20252 raft_consensus.cc:686] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 3 LEADER]: Becoming Leader. State: Replica: ab6acbd788e445179d8d625d71525849, State: Running, Role: LEADER
I20260812 19:11:19.276522 19898 raft_consensus.cc:2454] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 3.
I20260812 19:11:19.276917 20252 consensus_queue.cc:237] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:19.285465 19774 catalog_manager.cc:5495] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 reported cstate change: term changed from 2 to 3, leader changed from 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129) to ab6acbd788e445179d8d625d71525849 (127.19.70.131). New cstate: current_term: 3 leader_uuid: "ab6acbd788e445179d8d625d71525849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: HEALTHY } } }
I20260812 19:11:19.575640 19988 raft_consensus.cc:1261] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Refusing update from remote peer ab6acbd788e445179d8d625d71525849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:19.576843 20252 consensus_queue.cc:1035] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:11:19.587087 19898 raft_consensus.cc:1261] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Refusing update from remote peer ab6acbd788e445179d8d625d71525849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:19.588748 20252 consensus_queue.cc:1035] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:11:19.824081 19988 raft_consensus.cc:1261] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Refusing update from remote peer ab6acbd788e445179d8d625d71525849: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:19.825412 20252 consensus_queue.cc:1035] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:11:19.835947 19898 raft_consensus.cc:1261] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Refusing update from remote peer ab6acbd788e445179d8d625d71525849: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:19.837781 20252 consensus_queue.cc:1035] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:11:19.875665 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:19.880805 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:19.881285 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:20.882339 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:20.887840 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:20.888300 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:21.889113 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:21.894120 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:21.894598 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:22.895555 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:22.900784 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:22.901321 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:23.902256 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:23.907548 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:23.908025 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:24.908910 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:24.913678 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:24.914170 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:25.915084 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:25.920179 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:25.920634 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:26.921548 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:26.926332 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:26.926867 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:27.927794 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:27.933696 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:27.934157 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:28.969163 19738 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260812 19:11:28.974377 19895 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb783f78208046f586b7adb1ec2ea95a"
dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
mode: GRACEFUL
new_leader_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
from {username='slave'} at 127.0.0.1:44250
I20260812 19:11:28.975224 19895 raft_consensus.cc:595] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Received request to transfer leadership to TS 85cada24ba8d4fd8848da9cb03a8dd4b
I20260812 19:11:28.980844 19895 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "da6608b6ce0144edb54e3c4fa602c9cd"
dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
mode: GRACEFUL
new_leader_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
from {username='slave'} at 127.0.0.1:44250
I20260812 19:11:28.981406 19895 raft_consensus.cc:595] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Received request to transfer leadership to TS 85cada24ba8d4fd8848da9cb03a8dd4b
I20260812 19:11:28.986773 19895 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e"
dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
mode: GRACEFUL
new_leader_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
from {username='slave'} at 127.0.0.1:44250
I20260812 19:11:28.987363 19895 raft_consensus.cc:595] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Received request to transfer leadership to TS 85cada24ba8d4fd8848da9cb03a8dd4b
I20260812 19:11:28.992551 20075 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104"
dest_uuid: "ab6acbd788e445179d8d625d71525849"
mode: GRACEFUL
new_leader_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
from {username='slave'} at 127.0.0.1:53698
I20260812 19:11:28.993114 20075 raft_consensus.cc:595] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 3 LEADER]: Received request to transfer leadership to TS 85cada24ba8d4fd8848da9cb03a8dd4b
I20260812 19:11:28.998277 20075 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1521d1316ec944a5a50b839e70588916"
dest_uuid: "ab6acbd788e445179d8d625d71525849"
mode: GRACEFUL
new_leader_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
from {username='slave'} at 127.0.0.1:53698
I20260812 19:11:28.998924 20075 raft_consensus.cc:595] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 2 LEADER]: Received request to transfer leadership to TS 85cada24ba8d4fd8848da9cb03a8dd4b
I20260812 19:11:29.155717 20318 raft_consensus.cc:979] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577: : Instructing follower 85cada24ba8d4fd8848da9cb03a8dd4b to start an election
I20260812 19:11:29.156347 20245 raft_consensus.cc:1067] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Signalling peer 85cada24ba8d4fd8848da9cb03a8dd4b to start an election
I20260812 19:11:29.158026 19987 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb783f78208046f586b7adb1ec2ea95a"
dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
from {username='slave'} at 127.0.0.1:49234
I20260812 19:11:29.158601 19987 raft_consensus.cc:482] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:29.159097 19987 raft_consensus.cc:3037] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:29.162676 20318 raft_consensus.cc:979] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577: : Instructing follower 85cada24ba8d4fd8848da9cb03a8dd4b to start an election
I20260812 19:11:29.163260 20318 raft_consensus.cc:1067] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Signalling peer 85cada24ba8d4fd8848da9cb03a8dd4b to start an election
I20260812 19:11:29.165191 19988 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da6608b6ce0144edb54e3c4fa602c9cd"
dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
from {username='slave'} at 127.0.0.1:49234
I20260812 19:11:29.165786 19988 raft_consensus.cc:482] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:29.166159 19988 raft_consensus.cc:3037] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:29.167672 19987 raft_consensus.cc:504] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } }
I20260812 19:11:29.171098 19987 leader_election.cc:288] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 3 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:29.172144 20075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb783f78208046f586b7adb1ec2ea95a" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:29.172132 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb783f78208046f586b7adb1ec2ea95a" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:29.173082 20075 raft_consensus.cc:3037] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:29.173113 19895 raft_consensus.cc:3032] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:11:29.173609 19895 raft_consensus.cc:729] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Running, Role: LEADER
I20260812 19:11:29.173875 19988 raft_consensus.cc:504] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:29.174511 19895 consensus_queue.cc:260] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } }
I20260812 19:11:29.175685 19988 leader_election.cc:288] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 3 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:29.177111 19898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6608b6ce0144edb54e3c4fa602c9cd" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:29.177229 20077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6608b6ce0144edb54e3c4fa602c9cd" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:29.177748 19898 raft_consensus.cc:3032] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:11:29.177922 20077 raft_consensus.cc:3037] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:29.178083 19898 raft_consensus.cc:729] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Running, Role: LEADER
I20260812 19:11:29.178957 19898 consensus_queue.cc:260] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:29.179960 19898 raft_consensus.cc:3037] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:29.181891 20075 raft_consensus.cc:2454] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 3.
I20260812 19:11:29.176298 19895 raft_consensus.cc:3037] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:29.183511 19946 leader_election.cc:302] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:29.184665 20315 raft_consensus.cc:2781] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:11:29.186066 20077 raft_consensus.cc:2454] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 3.
I20260812 19:11:29.187582 19946 leader_election.cc:302] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:29.188403 20315 raft_consensus.cc:686] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 LEADER]: Becoming Leader. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Running, Role: LEADER
I20260812 19:11:29.188521 20319 raft_consensus.cc:2781] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:11:29.189851 19898 raft_consensus.cc:2454] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 3.
I20260812 19:11:29.189719 20315 consensus_queue.cc:237] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } }
I20260812 19:11:29.190593 19895 raft_consensus.cc:2454] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 3.
I20260812 19:11:29.193274 20319 raft_consensus.cc:686] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 LEADER]: Becoming Leader. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Running, Role: LEADER
I20260812 19:11:29.194550 20319 consensus_queue.cc:237] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:29.201504 19774 catalog_manager.cc:5495] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b reported cstate change: term changed from 2 to 3, leader changed from 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129) to 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130). New cstate: current_term: 3 leader_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: HEALTHY } } }
I20260812 19:11:29.220571 19774 catalog_manager.cc:5495] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b reported cstate change: term changed from 2 to 3, leader changed from 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129) to 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130). New cstate: current_term: 3 leader_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:29.339972 20252 raft_consensus.cc:979] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849: : Instructing follower 85cada24ba8d4fd8848da9cb03a8dd4b to start an election
I20260812 19:11:29.340418 20252 raft_consensus.cc:1067] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 3 LEADER]: Signalling peer 85cada24ba8d4fd8848da9cb03a8dd4b to start an election
I20260812 19:11:29.341876 19988 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104"
dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
from {username='slave'} at 127.0.0.1:49236
I20260812 19:11:29.342448 19988 raft_consensus.cc:482] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:29.342792 19988 raft_consensus.cc:3037] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:11:29.345589 20245 raft_consensus.cc:979] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577: : Instructing follower 85cada24ba8d4fd8848da9cb03a8dd4b to start an election
I20260812 19:11:29.346068 20318 raft_consensus.cc:1067] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Signalling peer 85cada24ba8d4fd8848da9cb03a8dd4b to start an election
I20260812 19:11:29.347693 19987 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e"
dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
from {username='slave'} at 127.0.0.1:49234
I20260812 19:11:29.348237 19987 raft_consensus.cc:482] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:29.348585 19987 raft_consensus.cc:3037] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:29.348869 19988 raft_consensus.cc:504] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:29.350797 19988 leader_election.cc:288] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 4 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:29.351552 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:29.351722 20075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f55ca1b3e840adbf76b36e23fae104" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:29.352128 19895 raft_consensus.cc:3037] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:11:29.352295 20075 raft_consensus.cc:3032] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:11:29.352694 20075 raft_consensus.cc:729] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab6acbd788e445179d8d625d71525849, State: Running, Role: LEADER
I20260812 19:11:29.353446 20075 consensus_queue.cc:260] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:29.355317 19987 raft_consensus.cc:504] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:29.357270 19987 leader_election.cc:288] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 3 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:29.357326 20075 raft_consensus.cc:3037] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:11:29.358879 19895 raft_consensus.cc:2454] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 4.
I20260812 19:11:29.358819 20077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:29.359589 20077 raft_consensus.cc:3037] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:29.359163 20252 raft_consensus.cc:979] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849: : Instructing follower 85cada24ba8d4fd8848da9cb03a8dd4b to start an election
I20260812 19:11:29.360201 20252 raft_consensus.cc:1067] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 2 LEADER]: Signalling peer 85cada24ba8d4fd8848da9cb03a8dd4b to start an election
I20260812 19:11:29.360535 19946 leader_election.cc:302] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, 85cada24ba8d4fd8848da9cb03a8dd4b; no voters:
I20260812 19:11:29.361330 20319 raft_consensus.cc:2781] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:11:29.361809 20319 raft_consensus.cc:686] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 4 LEADER]: Becoming Leader. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Running, Role: LEADER
I20260812 19:11:29.362005 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:29.362833 19895 raft_consensus.cc:3032] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:11:29.362751 20319 consensus_queue.cc:237] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:29.363306 19895 raft_consensus.cc:729] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Running, Role: LEADER
I20260812 19:11:29.364300 19895 consensus_queue.cc:260] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:29.364920 20075 raft_consensus.cc:2454] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 4.
I20260812 19:11:29.365747 19895 raft_consensus.cc:3037] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:29.367305 20077 raft_consensus.cc:2454] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 3.
I20260812 19:11:29.368650 19946 leader_election.cc:302] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:29.369493 20315 raft_consensus.cc:2781] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:11:29.369982 20315 raft_consensus.cc:686] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 LEADER]: Becoming Leader. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Running, Role: LEADER
I20260812 19:11:29.368316 19987 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1521d1316ec944a5a50b839e70588916"
dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
from {username='slave'} at 127.0.0.1:49236
I20260812 19:11:29.371035 20315 consensus_queue.cc:237] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:29.373023 19987 raft_consensus.cc:482] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:29.373057 19895 raft_consensus.cc:2454] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 3.
I20260812 19:11:29.373529 19987 raft_consensus.cc:3037] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:29.376567 19774 catalog_manager.cc:5495] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b reported cstate change: term changed from 3 to 4, leader changed from ab6acbd788e445179d8d625d71525849 (127.19.70.131) to 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130). New cstate: current_term: 4 leader_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:29.380209 19987 raft_consensus.cc:504] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:29.382184 19987 leader_election.cc:288] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 3 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:29.383003 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1521d1316ec944a5a50b839e70588916" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:29.383258 20077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1521d1316ec944a5a50b839e70588916" candidate_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:29.383602 19895 raft_consensus.cc:3037] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:29.383891 20077 raft_consensus.cc:3032] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:11:29.384213 20077 raft_consensus.cc:729] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ab6acbd788e445179d8d625d71525849, State: Running, Role: LEADER
I20260812 19:11:29.385041 20077 consensus_queue.cc:260] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:29.386257 20077 raft_consensus.cc:3037] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:29.389611 19895 raft_consensus.cc:2454] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 3.
I20260812 19:11:29.390676 19946 leader_election.cc:302] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, 85cada24ba8d4fd8848da9cb03a8dd4b; no voters:
I20260812 19:11:29.390492 19774 catalog_manager.cc:5495] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b reported cstate change: term changed from 2 to 3, leader changed from 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129) to 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130). New cstate: current_term: 3 leader_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:29.391407 20315 raft_consensus.cc:2781] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:11:29.391960 20315 raft_consensus.cc:686] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 LEADER]: Becoming Leader. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Running, Role: LEADER
I20260812 19:11:29.392779 20077 raft_consensus.cc:2454] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85cada24ba8d4fd8848da9cb03a8dd4b in term 3.
I20260812 19:11:29.392791 20315 consensus_queue.cc:237] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:29.403982 19774 catalog_manager.cc:5495] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b reported cstate change: term changed from 2 to 3, leader changed from ab6acbd788e445179d8d625d71525849 (127.19.70.131) to 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130). New cstate: current_term: 3 leader_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:29.609797 19895 raft_consensus.cc:1261] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Refusing update from remote peer 85cada24ba8d4fd8848da9cb03a8dd4b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:29.611330 20315 consensus_queue.cc:1035] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:11:29.630487 20077 raft_consensus.cc:1261] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Refusing update from remote peer 85cada24ba8d4fd8848da9cb03a8dd4b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:29.635360 20320 consensus_queue.cc:1035] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260812 19:11:29.735141 19895 raft_consensus.cc:1261] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Refusing update from remote peer 85cada24ba8d4fd8848da9cb03a8dd4b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:29.736366 20315 consensus_queue.cc:1035] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:11:29.752002 20077 raft_consensus.cc:1261] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Refusing update from remote peer 85cada24ba8d4fd8848da9cb03a8dd4b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:29.754509 20315 consensus_queue.cc:1035] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:11:29.762984 20075 raft_consensus.cc:1261] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Refusing update from remote peer 85cada24ba8d4fd8848da9cb03a8dd4b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:29.764820 20315 consensus_queue.cc:1035] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:11:29.776821 20077 raft_consensus.cc:1261] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 4 FOLLOWER]: Refusing update from remote peer 85cada24ba8d4fd8848da9cb03a8dd4b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:11:29.780404 20315 consensus_queue.cc:1035] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:11:29.783825 19895 raft_consensus.cc:1261] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Refusing update from remote peer 85cada24ba8d4fd8848da9cb03a8dd4b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:29.786873 20319 consensus_queue.cc:1035] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:11:29.806850 19895 raft_consensus.cc:1261] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 4 FOLLOWER]: Refusing update from remote peer 85cada24ba8d4fd8848da9cb03a8dd4b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:11:29.809690 20315 consensus_queue.cc:1035] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:11:29.816730 20077 raft_consensus.cc:1261] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Refusing update from remote peer 85cada24ba8d4fd8848da9cb03a8dd4b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:29.818871 20319 consensus_queue.cc:1035] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:11:29.818807 19895 raft_consensus.cc:1261] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Refusing update from remote peer 85cada24ba8d4fd8848da9cb03a8dd4b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:29.830737 20319 consensus_queue.cc:1035] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260812 19:11:32.042511 19738 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260812 19:11:32.043372 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:32.050060 19987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1521d1316ec944a5a50b839e70588916"
dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
mode: GRACEFUL
new_leader_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
from {username='slave'} at 127.0.0.1:53186
I20260812 19:11:32.050668 19987 raft_consensus.cc:595] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 LEADER]: Received request to transfer leadership to TS 3448dd2b5d4b4a598f259f591c5b3577
I20260812 19:11:32.053642 19987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e475f91eeb94e0b9bfbd134a1da96fe"
dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
mode: GRACEFUL
new_leader_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
from {username='slave'} at 127.0.0.1:53186
I20260812 19:11:32.054322 19987 raft_consensus.cc:595] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 LEADER]: Received request to transfer leadership to TS 3448dd2b5d4b4a598f259f591c5b3577
I20260812 19:11:32.057065 19987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "90a2c3fe70be421ba0639d24fabe55ca"
dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
mode: GRACEFUL
new_leader_uuid: "ab6acbd788e445179d8d625d71525849"
from {username='slave'} at 127.0.0.1:53186
I20260812 19:11:32.057590 19987 raft_consensus.cc:595] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 LEADER]: Received request to transfer leadership to TS ab6acbd788e445179d8d625d71525849
I20260812 19:11:32.059854 19987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e"
dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
mode: GRACEFUL
new_leader_uuid: "ab6acbd788e445179d8d625d71525849"
from {username='slave'} at 127.0.0.1:53186
I20260812 19:11:32.060372 19987 raft_consensus.cc:595] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 LEADER]: Received request to transfer leadership to TS ab6acbd788e445179d8d625d71525849
I20260812 19:11:32.062732 19987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0e1f458ea8c4061adcdb488b7c02c2f"
dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
mode: GRACEFUL
new_leader_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
from {username='slave'} at 127.0.0.1:53186
I20260812 19:11:32.063301 19987 raft_consensus.cc:595] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 LEADER]: Received request to transfer leadership to TS 3448dd2b5d4b4a598f259f591c5b3577
I20260812 19:11:32.064467 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260812 19:11:32.064950 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:32.220700 20315 raft_consensus.cc:979] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b: : Instructing follower 3448dd2b5d4b4a598f259f591c5b3577 to start an election
I20260812 19:11:32.221216 20351 raft_consensus.cc:1067] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 LEADER]: Signalling peer 3448dd2b5d4b4a598f259f591c5b3577 to start an election
I20260812 19:11:32.222754 19898 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1521d1316ec944a5a50b839e70588916"
dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
from {username='slave'} at 127.0.0.1:58164
I20260812 19:11:32.223263 19898 raft_consensus.cc:482] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:32.223568 19898 raft_consensus.cc:3037] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:11:32.230079 19898 raft_consensus.cc:504] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:32.232726 19898 leader_election.cc:288] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 4 election: Requested vote from peers 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:32.233837 20075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1521d1316ec944a5a50b839e70588916" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:32.233945 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1521d1316ec944a5a50b839e70588916" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:32.234536 20075 raft_consensus.cc:3037] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:11:32.234602 19987 raft_consensus.cc:3032] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:11:32.235035 19987 raft_consensus.cc:729] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Running, Role: LEADER
I20260812 19:11:32.235775 19987 consensus_queue.cc:260] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:32.237044 19987 raft_consensus.cc:3037] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:11:32.239785 20075 raft_consensus.cc:2454] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 4.
I20260812 19:11:32.241230 19857 leader_election.cc:302] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:32.242169 19987 raft_consensus.cc:2454] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 4.
I20260812 19:11:32.243186 20354 raft_consensus.cc:2781] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:11:32.244570 20354 raft_consensus.cc:686] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 4 LEADER]: Becoming Leader. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Running, Role: LEADER
I20260812 19:11:32.245426 20354 consensus_queue.cc:237] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:32.253060 19774 catalog_manager.cc:5495] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 reported cstate change: term changed from 3 to 4, leader changed from 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130) to 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129). New cstate: current_term: 4 leader_uuid: "3448dd2b5d4b4a598f259f591c5b3577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:32.327877 20315 raft_consensus.cc:979] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b: : Instructing follower 3448dd2b5d4b4a598f259f591c5b3577 to start an election
I20260812 19:11:32.328372 20315 raft_consensus.cc:1067] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 LEADER]: Signalling peer 3448dd2b5d4b4a598f259f591c5b3577 to start an election
I20260812 19:11:32.329811 19898 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0e1f458ea8c4061adcdb488b7c02c2f"
dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
from {username='slave'} at 127.0.0.1:58164
I20260812 19:11:32.330349 19898 raft_consensus.cc:482] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:32.330708 19898 raft_consensus.cc:3037] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:32.336475 19898 raft_consensus.cc:504] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:32.338294 19898 leader_election.cc:288] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 3 election: Requested vote from peers 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:32.339154 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1f458ea8c4061adcdb488b7c02c2f" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:32.339475 20075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1f458ea8c4061adcdb488b7c02c2f" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:32.339891 20315 raft_consensus.cc:979] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b: : Instructing follower 3448dd2b5d4b4a598f259f591c5b3577 to start an election
I20260812 19:11:32.339797 19987 raft_consensus.cc:3032] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:11:32.340037 20075 raft_consensus.cc:3037] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:32.340250 19987 raft_consensus.cc:729] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Running, Role: LEADER
I20260812 19:11:32.340380 20351 raft_consensus.cc:1067] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 LEADER]: Signalling peer 3448dd2b5d4b4a598f259f591c5b3577 to start an election
I20260812 19:11:32.341151 19987 consensus_queue.cc:260] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:32.342432 19987 raft_consensus.cc:3037] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:32.343142 19898 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e475f91eeb94e0b9bfbd134a1da96fe"
dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
from {username='slave'} at 127.0.0.1:58164
I20260812 19:11:32.343745 19898 raft_consensus.cc:482] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:32.344184 19898 raft_consensus.cc:3037] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:32.347067 20075 raft_consensus.cc:2454] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 3.
I20260812 19:11:32.347575 19987 raft_consensus.cc:2454] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 3.
I20260812 19:11:32.348259 19857 leader_election.cc:302] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:32.349169 20354 raft_consensus.cc:2781] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:11:32.349612 20354 raft_consensus.cc:686] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 LEADER]: Becoming Leader. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Running, Role: LEADER
I20260812 19:11:32.350517 19898 raft_consensus.cc:504] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:32.350446 20354 consensus_queue.cc:237] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:32.352289 19898 leader_election.cc:288] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 2 election: Requested vote from peers 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501), ab6acbd788e445179d8d625d71525849 (127.19.70.131:43037)
I20260812 19:11:32.353497 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e475f91eeb94e0b9bfbd134a1da96fe" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:32.354436 19987 raft_consensus.cc:3032] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:11:32.354745 19987 raft_consensus.cc:729] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Running, Role: LEADER
I20260812 19:11:32.355475 19987 consensus_queue.cc:260] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:32.353514 20075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e475f91eeb94e0b9bfbd134a1da96fe" candidate_uuid: "3448dd2b5d4b4a598f259f591c5b3577" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab6acbd788e445179d8d625d71525849"
I20260812 19:11:32.356323 20075 raft_consensus.cc:3037] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:32.356482 19987 raft_consensus.cc:3037] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:32.359484 19774 catalog_manager.cc:5495] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 reported cstate change: term changed from 2 to 3, leader changed from 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130) to 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "3448dd2b5d4b4a598f259f591c5b3577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:32.362135 20351 raft_consensus.cc:979] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b: : Instructing follower ab6acbd788e445179d8d625d71525849 to start an election
I20260812 19:11:32.362686 20315 raft_consensus.cc:1067] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 LEADER]: Signalling peer ab6acbd788e445179d8d625d71525849 to start an election
I20260812 19:11:32.363140 19987 raft_consensus.cc:2454] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 2.
I20260812 19:11:32.363593 20075 raft_consensus.cc:2454] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3448dd2b5d4b4a598f259f591c5b3577 in term 2.
I20260812 19:11:32.364508 19853 leader_election.cc:302] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, 85cada24ba8d4fd8848da9cb03a8dd4b; no voters:
I20260812 19:11:32.364841 20077 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90a2c3fe70be421ba0639d24fabe55ca"
dest_uuid: "ab6acbd788e445179d8d625d71525849"
from {username='slave'} at 127.0.0.1:33884
I20260812 19:11:32.365420 20077 raft_consensus.cc:482] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:32.365713 20354 raft_consensus.cc:2781] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:11:32.365808 20077 raft_consensus.cc:3037] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:32.366261 20354 raft_consensus.cc:686] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Becoming Leader. State: Replica: 3448dd2b5d4b4a598f259f591c5b3577, State: Running, Role: LEADER
I20260812 19:11:32.367180 20354 consensus_queue.cc:237] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:32.373452 20077 raft_consensus.cc:504] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:32.377003 20077 leader_election.cc:288] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 3 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:32.377853 19898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a2c3fe70be421ba0639d24fabe55ca" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:32.378078 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a2c3fe70be421ba0639d24fabe55ca" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:32.378518 19898 raft_consensus.cc:3037] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:32.380506 19774 catalog_manager.cc:5495] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 reported cstate change: term changed from 1 to 2, leader changed from 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130) to 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129). New cstate: current_term: 2 leader_uuid: "3448dd2b5d4b4a598f259f591c5b3577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:32.378543 19987 raft_consensus.cc:3032] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:11:32.381819 19987 raft_consensus.cc:729] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Running, Role: LEADER
I20260812 19:11:32.382575 19987 consensus_queue.cc:260] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:32.383896 19987 raft_consensus.cc:3037] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:32.385002 19898 raft_consensus.cc:2454] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 3.
I20260812 19:11:32.386394 20037 leader_election.cc:302] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3448dd2b5d4b4a598f259f591c5b3577, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:32.387327 20331 raft_consensus.cc:2781] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:11:32.388089 20351 raft_consensus.cc:979] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b: : Instructing follower ab6acbd788e445179d8d625d71525849 to start an election
I20260812 19:11:32.388597 20315 raft_consensus.cc:1067] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 LEADER]: Signalling peer ab6acbd788e445179d8d625d71525849 to start an election
I20260812 19:11:32.390244 20077 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e"
dest_uuid: "ab6acbd788e445179d8d625d71525849"
from {username='slave'} at 127.0.0.1:33884
I20260812 19:11:32.390565 20331 raft_consensus.cc:686] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 3 LEADER]: Becoming Leader. State: Replica: ab6acbd788e445179d8d625d71525849, State: Running, Role: LEADER
I20260812 19:11:32.391037 20077 raft_consensus.cc:482] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:32.391474 20077 raft_consensus.cc:3037] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:11:32.392287 20331 consensus_queue.cc:237] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:32.396109 19987 raft_consensus.cc:2454] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 3.
I20260812 19:11:32.398371 20077 raft_consensus.cc:504] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:32.400794 20077 leader_election.cc:288] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 4 election: Requested vote from peers 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063), 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501)
I20260812 19:11:32.401408 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b"
I20260812 19:11:32.402038 19987 raft_consensus.cc:3032] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:11:32.402333 19987 raft_consensus.cc:729] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85cada24ba8d4fd8848da9cb03a8dd4b, State: Running, Role: LEADER
I20260812 19:11:32.402977 19898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c08be155824b58aa8242e0aa68a62e" candidate_uuid: "ab6acbd788e445179d8d625d71525849" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3448dd2b5d4b4a598f259f591c5b3577"
I20260812 19:11:32.403487 19898 raft_consensus.cc:3037] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:11:32.403053 19987 consensus_queue.cc:260] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:32.404351 19987 raft_consensus.cc:3037] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:11:32.403915 19774 catalog_manager.cc:5495] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 reported cstate change: term changed from 2 to 3, leader changed from 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130) to ab6acbd788e445179d8d625d71525849 (127.19.70.131). New cstate: current_term: 3 leader_uuid: "ab6acbd788e445179d8d625d71525849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: HEALTHY } } }
I20260812 19:11:32.409389 19987 raft_consensus.cc:2454] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 4.
I20260812 19:11:32.409600 19898 raft_consensus.cc:2454] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6acbd788e445179d8d625d71525849 in term 4.
I20260812 19:11:32.410583 20033 leader_election.cc:302] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85cada24ba8d4fd8848da9cb03a8dd4b, ab6acbd788e445179d8d625d71525849; no voters:
I20260812 19:11:32.411453 20331 raft_consensus.cc:2781] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:11:32.411873 20331 raft_consensus.cc:686] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 4 LEADER]: Becoming Leader. State: Replica: ab6acbd788e445179d8d625d71525849, State: Running, Role: LEADER
I20260812 19:11:32.412535 20331 consensus_queue.cc:237] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } }
I20260812 19:11:32.419963 19774 catalog_manager.cc:5495] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 reported cstate change: term changed from 3 to 4, leader changed from 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130) to ab6acbd788e445179d8d625d71525849 (127.19.70.131). New cstate: current_term: 4 leader_uuid: "ab6acbd788e445179d8d625d71525849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 } health_report { overall_health: HEALTHY } } }
I20260812 19:11:32.685133 20077 raft_consensus.cc:1261] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 4 FOLLOWER]: Refusing update from remote peer 3448dd2b5d4b4a598f259f591c5b3577: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:11:32.686448 20354 consensus_queue.cc:1035] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:11:32.697841 19987 raft_consensus.cc:1261] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 4 FOLLOWER]: Refusing update from remote peer 3448dd2b5d4b4a598f259f591c5b3577: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:11:32.699206 20354 consensus_queue.cc:1035] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:11:32.744760 20077 raft_consensus.cc:1261] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Refusing update from remote peer 3448dd2b5d4b4a598f259f591c5b3577: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:32.746202 20354 consensus_queue.cc:1035] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:11:32.755664 19987 raft_consensus.cc:1261] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Refusing update from remote peer 3448dd2b5d4b4a598f259f591c5b3577: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:32.757359 20354 consensus_queue.cc:1035] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:11:32.772380 20077 raft_consensus.cc:1261] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Refusing update from remote peer 3448dd2b5d4b4a598f259f591c5b3577: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:32.773727 20354 consensus_queue.cc:1035] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6acbd788e445179d8d625d71525849" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:11:32.783046 19987 raft_consensus.cc:1261] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Refusing update from remote peer 3448dd2b5d4b4a598f259f591c5b3577: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:32.784590 20378 consensus_queue.cc:1035] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:32.803565 19898 raft_consensus.cc:1261] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Refusing update from remote peer ab6acbd788e445179d8d625d71525849: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:32.804831 20331 consensus_queue.cc:1035] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:11:32.814849 19987 raft_consensus.cc:1261] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Refusing update from remote peer ab6acbd788e445179d8d625d71525849: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:32.816649 20366 consensus_queue.cc:1035] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:11:32.944118 19987 raft_consensus.cc:1261] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 4 FOLLOWER]: Refusing update from remote peer ab6acbd788e445179d8d625d71525849: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:11:32.945457 20366 consensus_queue.cc:1035] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cada24ba8d4fd8848da9cb03a8dd4b" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:11:32.954600 19898 raft_consensus.cc:1261] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 4 FOLLOWER]: Refusing update from remote peer ab6acbd788e445179d8d625d71525849: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:11:32.957005 20384 consensus_queue.cc:1035] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3448dd2b5d4b4a598f259f591c5b3577" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:11:33.065855 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:33.072196 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:33.072597 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:34.073470 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:34.079205 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:34.079643 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:35.080590 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:35.085857 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:35.086339 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:36.087263 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:36.092972 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:36.093500 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:37.094473 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:37.099339 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:37.099838 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:38.100797 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:38.107300 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:38.107795 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:39.108836 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:39.114713 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:39.115219 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:40.116626 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:40.122287 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:40.122867 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:41.123694 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:41.128798 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:41.129225 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:42.170853 19738 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260812 19:11:42.171797 19738 tablet_server.cc:178] TabletServer@127.19.70.129:0 shutting down...
W20260812 19:11:42.210911 19946 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260812 19:11:42.215027 19738 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260812 19:11:42.215996 19738 tablet_replica.cc:330] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577: stopping tablet replica
W20260812 19:11:42.217120 19946 consensus_peers.cc:487] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b -> Peer 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063): Couldn't send request to peer 3448dd2b5d4b4a598f259f591c5b3577. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:11:42.217226 19738 raft_consensus.cc:2229] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 LEADER]: Raft consensus shutting down.
I20260812 19:11:42.218736 19738 raft_consensus.cc:2258] T 4e475f91eeb94e0b9bfbd134a1da96fe P 3448dd2b5d4b4a598f259f591c5b3577 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.221354 19738 tablet_replica.cc:330] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577: stopping tablet replica
I20260812 19:11:42.221933 19738 raft_consensus.cc:2229] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.222452 19738 raft_consensus.cc:2258] T 90a2c3fe70be421ba0639d24fabe55ca P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.224301 19738 tablet_replica.cc:330] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577: stopping tablet replica
I20260812 19:11:42.224846 19738 raft_consensus.cc:2229] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.225347 19738 raft_consensus.cc:2258] T 97c08be155824b58aa8242e0aa68a62e P 3448dd2b5d4b4a598f259f591c5b3577 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.227456 19738 tablet_replica.cc:330] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577: stopping tablet replica
I20260812 19:11:42.228003 19738 raft_consensus.cc:2229] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.228473 19738 raft_consensus.cc:2258] T fb783f78208046f586b7adb1ec2ea95a P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.230172 19738 tablet_replica.cc:330] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577: stopping tablet replica
I20260812 19:11:42.230826 19738 raft_consensus.cc:2229] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.231379 19738 raft_consensus.cc:2258] T 40f55ca1b3e840adbf76b36e23fae104 P 3448dd2b5d4b4a598f259f591c5b3577 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.233378 19738 tablet_replica.cc:330] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577: stopping tablet replica
I20260812 19:11:42.233965 19738 raft_consensus.cc:2229] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.234474 19738 raft_consensus.cc:2258] T da6608b6ce0144edb54e3c4fa602c9cd P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.236322 19738 tablet_replica.cc:330] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577: stopping tablet replica
I20260812 19:11:42.236899 19738 raft_consensus.cc:2229] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.237408 19738 raft_consensus.cc:2258] T 61c8a97dda724773a4bb69e23875e844 P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.239380 19738 tablet_replica.cc:330] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577: stopping tablet replica
I20260812 19:11:42.239902 19738 raft_consensus.cc:2229] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 4 LEADER]: Raft consensus shutting down.
I20260812 19:11:42.240862 19738 raft_consensus.cc:2258] T 1521d1316ec944a5a50b839e70588916 P 3448dd2b5d4b4a598f259f591c5b3577 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.242573 19738 tablet_replica.cc:330] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577: stopping tablet replica
I20260812 19:11:42.243170 19738 raft_consensus.cc:2229] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 LEADER]: Raft consensus shutting down.
I20260812 19:11:42.244067 19738 raft_consensus.cc:2258] T d0e1f458ea8c4061adcdb488b7c02c2f P 3448dd2b5d4b4a598f259f591c5b3577 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.348837 19738 tablet_server.cc:194] TabletServer@127.19.70.129:0 shutdown complete.
I20260812 19:11:42.370437 19738 tablet_server.cc:178] TabletServer@127.19.70.130:0 shutting down...
I20260812 19:11:42.407603 19738 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260812 19:11:42.408362 19738 tablet_replica.cc:330] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b: stopping tablet replica
I20260812 19:11:42.409085 19738 raft_consensus.cc:2229] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.409627 19738 raft_consensus.cc:2258] T 61c8a97dda724773a4bb69e23875e844 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.411763 19738 tablet_replica.cc:330] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b: stopping tablet replica
I20260812 19:11:42.412427 19738 raft_consensus.cc:2229] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.412950 19738 raft_consensus.cc:2258] T d0e1f458ea8c4061adcdb488b7c02c2f P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.414974 19738 tablet_replica.cc:330] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b: stopping tablet replica
I20260812 19:11:42.415520 19738 raft_consensus.cc:2229] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.416016 19738 raft_consensus.cc:2258] T 1521d1316ec944a5a50b839e70588916 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.417850 19738 tablet_replica.cc:330] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b: stopping tablet replica
I20260812 19:11:42.418396 19738 raft_consensus.cc:2229] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 4 LEADER]: Raft consensus shutting down.
I20260812 19:11:42.419689 19738 raft_consensus.cc:2258] T 40f55ca1b3e840adbf76b36e23fae104 P 85cada24ba8d4fd8848da9cb03a8dd4b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.421861 19738 tablet_replica.cc:330] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b: stopping tablet replica
I20260812 19:11:42.422499 19738 raft_consensus.cc:2229] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 LEADER]: Raft consensus shutting down.
I20260812 19:11:42.423497 19738 raft_consensus.cc:2258] T fb783f78208046f586b7adb1ec2ea95a P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.425387 19738 tablet_replica.cc:330] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b: stopping tablet replica
I20260812 19:11:42.425932 19738 raft_consensus.cc:2229] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 LEADER]: Raft consensus shutting down.
I20260812 19:11:42.426925 19738 raft_consensus.cc:2258] T da6608b6ce0144edb54e3c4fa602c9cd P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.428668 19738 tablet_replica.cc:330] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b: stopping tablet replica
I20260812 19:11:42.429215 19738 raft_consensus.cc:2229] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.429703 19738 raft_consensus.cc:2258] T 97c08be155824b58aa8242e0aa68a62e P 85cada24ba8d4fd8848da9cb03a8dd4b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.431680 19738 tablet_replica.cc:330] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b: stopping tablet replica
I20260812 19:11:42.432220 19738 raft_consensus.cc:2229] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.432760 19738 raft_consensus.cc:2258] T 90a2c3fe70be421ba0639d24fabe55ca P 85cada24ba8d4fd8848da9cb03a8dd4b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.434871 19738 tablet_replica.cc:330] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b: stopping tablet replica
I20260812 19:11:42.435374 19738 raft_consensus.cc:2229] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.435874 19738 raft_consensus.cc:2258] T 4e475f91eeb94e0b9bfbd134a1da96fe P 85cada24ba8d4fd8848da9cb03a8dd4b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.468823 19738 tablet_server.cc:194] TabletServer@127.19.70.130:0 shutdown complete.
I20260812 19:11:42.489740 19738 tablet_server.cc:178] TabletServer@127.19.70.131:0 shutting down...
W20260812 19:11:42.517710 20037 consensus_peers.cc:487] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 -> Peer 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063): Couldn't send request to peer 3448dd2b5d4b4a598f259f591c5b3577. Status: Network error: Client connection negotiation failed: client connection to 127.19.70.129:34063: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 19:11:42.525415 20037 consensus_peers.cc:487] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 -> Peer 3448dd2b5d4b4a598f259f591c5b3577 (127.19.70.129:34063): Couldn't send request to peer 3448dd2b5d4b4a598f259f591c5b3577. Status: Network error: Client connection negotiation failed: client connection to 127.19.70.129:34063: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 19:11:42.529254 20033 consensus_peers.cc:487] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 -> Peer 85cada24ba8d4fd8848da9cb03a8dd4b (127.19.70.130:40501): Couldn't send request to peer 85cada24ba8d4fd8848da9cb03a8dd4b. Status: Network error: Client connection negotiation failed: client connection to 127.19.70.130:40501: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 19:11:42.530681 19738 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260812 19:11:42.531327 19738 tablet_replica.cc:330] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849: stopping tablet replica
I20260812 19:11:42.531993 19738 raft_consensus.cc:2229] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 4 LEADER]: Raft consensus shutting down.
I20260812 19:11:42.532878 19738 raft_consensus.cc:2258] T 97c08be155824b58aa8242e0aa68a62e P ab6acbd788e445179d8d625d71525849 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.534899 19738 tablet_replica.cc:330] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849: stopping tablet replica
I20260812 19:11:42.535457 19738 raft_consensus.cc:2229] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 3 LEADER]: Raft consensus shutting down.
I20260812 19:11:42.536321 19738 raft_consensus.cc:2258] T 90a2c3fe70be421ba0639d24fabe55ca P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.538049 19738 tablet_replica.cc:330] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849: stopping tablet replica
I20260812 19:11:42.538575 19738 raft_consensus.cc:2229] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.539127 19738 raft_consensus.cc:2258] T da6608b6ce0144edb54e3c4fa602c9cd P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.540932 19738 tablet_replica.cc:330] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849: stopping tablet replica
I20260812 19:11:42.541447 19738 raft_consensus.cc:2229] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.541936 19738 raft_consensus.cc:2258] T 4e475f91eeb94e0b9bfbd134a1da96fe P ab6acbd788e445179d8d625d71525849 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.543712 19738 tablet_replica.cc:330] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849: stopping tablet replica
I20260812 19:11:42.544339 19738 raft_consensus.cc:2229] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.544850 19738 raft_consensus.cc:2258] T d0e1f458ea8c4061adcdb488b7c02c2f P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.546746 19738 tablet_replica.cc:330] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849: stopping tablet replica
I20260812 19:11:42.547246 19738 raft_consensus.cc:2229] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.547732 19738 raft_consensus.cc:2258] T 1521d1316ec944a5a50b839e70588916 P ab6acbd788e445179d8d625d71525849 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.549405 19738 tablet_replica.cc:330] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849: stopping tablet replica
I20260812 19:11:42.549892 19738 raft_consensus.cc:2229] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 3 LEADER]: Raft consensus shutting down.
I20260812 19:11:42.550741 19738 raft_consensus.cc:2258] T 61c8a97dda724773a4bb69e23875e844 P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.552515 19738 tablet_replica.cc:330] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849: stopping tablet replica
I20260812 19:11:42.553030 19738 raft_consensus.cc:2229] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.553465 19738 raft_consensus.cc:2258] T 40f55ca1b3e840adbf76b36e23fae104 P ab6acbd788e445179d8d625d71525849 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.555145 19738 tablet_replica.cc:330] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849: stopping tablet replica
I20260812 19:11:42.555615 19738 raft_consensus.cc:2229] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260812 19:11:42.556030 19738 raft_consensus.cc:2258] T fb783f78208046f586b7adb1ec2ea95a P ab6acbd788e445179d8d625d71525849 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.586524 19738 tablet_server.cc:194] TabletServer@127.19.70.131:0 shutdown complete.
I20260812 19:11:42.606793 19738 master.cc:537] Master@127.19.70.190:46041 shutting down...
I20260812 19:11:42.790453 19738 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [term 1 LEADER]: Raft consensus shutting down.
I20260812 19:11:42.791245 19738 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:11:42.791640 19738 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4f3e629f8dc34a6c86f47792c7ff5c24: stopping tablet replica
I20260812 19:11:42.812095 19738 master.cc:552] Master@127.19.70.190:46041 shutdown complete.
3448dd2b5d4b4a598f259f591c5b3577 4
85cada24ba8d4fd8848da9cb03a8dd4b 4
ab6acbd788e445179d8d625d71525849 1
3448dd2b5d4b4a598f259f591c5b3577 3
85cada24ba8d4fd8848da9cb03a8dd4b 3
ab6acbd788e445179d8d625d71525849 3
3448dd2b5d4b4a598f259f591c5b3577 0
85cada24ba8d4fd8848da9cb03a8dd4b 8
ab6acbd788e445179d8d625d71525849 1
3448dd2b5d4b4a598f259f591c5b3577 3
85cada24ba8d4fd8848da9cb03a8dd4b 3
ab6acbd788e445179d8d625d71525849 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (34542 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260812 19:11:42.859144 19738 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.70.190:45491
I20260812 19:11:42.860291 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 19:11:42.866545 20395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:11:42.867460 20396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:11:42.868695 19738 server_base.cc:1034] running on GCE node
W20260812 19:11:42.868638 20398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:11:42.870275 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:11:42.870481 19738 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:11:42.870744 19738 hybrid_clock.cc:648] HybridClock initialized: now 1786561902870727 us; error 0 us; skew 500 ppm
I20260812 19:11:42.871333 19738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:11:42.879254 19738 webserver.cc:460] Webserver started at http://127.19.70.190:41839/ using document root <none> and password file <none>
I20260812 19:11:42.879844 19738 fs_manager.cc:359] Metadata directory not provided
I20260812 19:11:42.880051 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:11:42.880386 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:11:42.881763 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/master-0-root/instance:
uuid: "68688543ce6f465380d67a94afa60e38"
format_stamp: "Formatted at 2026-08-12 19:11:42 on dist-test-slave-pgkr"
I20260812 19:11:42.886893 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260812 19:11:42.890753 20403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:42.891556 19738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 19:11:42.891891 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/master-0-root
uuid: "68688543ce6f465380d67a94afa60e38"
format_stamp: "Formatted at 2026-08-12 19:11:42 on dist-test-slave-pgkr"
I20260812 19:11:42.892174 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:11:42.920183 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 19:11:42.921591 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:11:42.966274 19738 rpc_server.cc:307] RPC server started. Bound to: 127.19.70.190:45491
I20260812 19:11:42.966413 20467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.70.190:45491 every 8 connection(s)
I20260812 19:11:42.970702 20468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:42.981477 20468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38: Bootstrap starting.
I20260812 19:11:42.986266 20468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:42.991153 20468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38: No bootstrap required, opened a new log
I20260812 19:11:42.993667 20468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68688543ce6f465380d67a94afa60e38" member_type: VOTER }
I20260812 19:11:42.994333 20468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:42.994550 20468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68688543ce6f465380d67a94afa60e38, State: Initialized, Role: FOLLOWER
I20260812 19:11:42.995183 20468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68688543ce6f465380d67a94afa60e38" member_type: VOTER }
I20260812 19:11:42.995657 20468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 19:11:42.995855 20468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 19:11:42.996074 20468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:43.000469 20468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68688543ce6f465380d67a94afa60e38" member_type: VOTER }
I20260812 19:11:43.001026 20468 leader_election.cc:302] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68688543ce6f465380d67a94afa60e38; no voters:
I20260812 19:11:43.002235 20468 leader_election.cc:288] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 19:11:43.002818 20474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:43.004340 20474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [term 1 LEADER]: Becoming Leader. State: Replica: 68688543ce6f465380d67a94afa60e38, State: Running, Role: LEADER
I20260812 19:11:43.005165 20474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68688543ce6f465380d67a94afa60e38" member_type: VOTER }
I20260812 19:11:43.005748 20468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:11:43.012385 20479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68688543ce6f465380d67a94afa60e38. Latest consensus state: current_term: 1 leader_uuid: "68688543ce6f465380d67a94afa60e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68688543ce6f465380d67a94afa60e38" member_type: VOTER } }
I20260812 19:11:43.014111 20479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [sys.catalog]: This master's current role is: LEADER
I20260812 19:11:43.014791 20477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68688543ce6f465380d67a94afa60e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68688543ce6f465380d67a94afa60e38" member_type: VOTER } }
I20260812 19:11:43.015447 20477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [sys.catalog]: This master's current role is: LEADER
I20260812 19:11:43.016625 20487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 19:11:43.022493 20487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 19:11:43.024768 19738 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260812 19:11:43.032886 20487 catalog_manager.cc:1347] Generated new cluster ID: ab9741cc2e214d6394b9c8b167229769
I20260812 19:11:43.033272 20487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 19:11:43.059870 20487 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 19:11:43.061841 20487 catalog_manager.cc:1504] Loading token signing keys...
I20260812 19:11:43.074992 20487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38: Generated new TSK 0
I20260812 19:11:43.075727 20487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 19:11:43.092208 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 19:11:43.099462 20496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:11:43.101016 20497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:11:43.102659 20499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:11:43.102931 19738 server_base.cc:1034] running on GCE node
I20260812 19:11:43.104149 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:11:43.104423 19738 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:11:43.104633 19738 hybrid_clock.cc:648] HybridClock initialized: now 1786561903104603 us; error 0 us; skew 500 ppm
I20260812 19:11:43.105232 19738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:11:43.108174 19738 webserver.cc:460] Webserver started at http://127.19.70.129:34317/ using document root <none> and password file <none>
I20260812 19:11:43.108780 19738 fs_manager.cc:359] Metadata directory not provided
I20260812 19:11:43.109018 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:11:43.109318 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:11:43.110659 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/ts-0-root/instance:
uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
format_stamp: "Formatted at 2026-08-12 19:11:43 on dist-test-slave-pgkr"
I20260812 19:11:43.115958 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260812 19:11:43.119954 20508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:43.120858 19738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260812 19:11:43.121201 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/ts-0-root
uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
format_stamp: "Formatted at 2026-08-12 19:11:43 on dist-test-slave-pgkr"
I20260812 19:11:43.121562 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:11:43.160147 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 19:11:43.161490 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:11:43.163430 19738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 19:11:43.167479 19738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:11:43.167706 19738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:43.168020 19738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 19:11:43.168211 19738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:43.213531 19738 rpc_server.cc:307] RPC server started. Bound to: 127.19.70.129:35187
I20260812 19:11:43.213660 20578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.70.129:35187 every 8 connection(s)
I20260812 19:11:43.218827 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 19:11:43.226436 20583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:11:43.228695 20584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:11:43.230860 20587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:11:43.232575 19738 server_base.cc:1034] running on GCE node
I20260812 19:11:43.234745 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:11:43.235049 19738 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:11:43.235234 19738 hybrid_clock.cc:648] HybridClock initialized: now 1786561903235220 us; error 0 us; skew 500 ppm
I20260812 19:11:43.235585 20579 heartbeater.cc:347] Connected to a master server at 127.19.70.190:45491
I20260812 19:11:43.235812 19738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:11:43.235968 20579 heartbeater.cc:464] Registering TS with master...
I20260812 19:11:43.236917 20579 heartbeater.cc:511] Master 127.19.70.190:45491 requested a full tablet report, sending...
I20260812 19:11:43.239100 19738 webserver.cc:460] Webserver started at http://127.19.70.130:38819/ using document root <none> and password file <none>
I20260812 19:11:43.239214 20423 ts_manager.cc:194] Registered new tserver with Master: 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187)
I20260812 19:11:43.239727 19738 fs_manager.cc:359] Metadata directory not provided
I20260812 19:11:43.239970 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:11:43.240254 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:11:43.241385 20423 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38870
I20260812 19:11:43.241437 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/ts-1-root/instance:
uuid: "c9645294d712440283be14a35f4e4beb"
format_stamp: "Formatted at 2026-08-12 19:11:43 on dist-test-slave-pgkr"
I20260812 19:11:43.247126 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 19:11:43.251714 20592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:43.252859 19738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 19:11:43.253280 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/ts-1-root
uuid: "c9645294d712440283be14a35f4e4beb"
format_stamp: "Formatted at 2026-08-12 19:11:43 on dist-test-slave-pgkr"
I20260812 19:11:43.253675 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:11:43.271893 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 19:11:43.273175 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:11:43.274776 19738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 19:11:43.277289 19738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:11:43.277518 19738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:43.277788 19738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 19:11:43.277943 19738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:43.321261 19738 rpc_server.cc:307] RPC server started. Bound to: 127.19.70.130:39101
I20260812 19:11:43.321378 20663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.70.130:39101 every 8 connection(s)
I20260812 19:11:43.326494 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 19:11:43.334340 20667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:11:43.335179 20668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:11:43.338269 19738 server_base.cc:1034] running on GCE node
W20260812 19:11:43.338198 20672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:11:43.339679 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:11:43.339968 19738 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:11:43.340199 19738 hybrid_clock.cc:648] HybridClock initialized: now 1786561903340167 us; error 0 us; skew 500 ppm
I20260812 19:11:43.340912 19738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:11:43.343029 20664 heartbeater.cc:347] Connected to a master server at 127.19.70.190:45491
I20260812 19:11:43.343446 20664 heartbeater.cc:464] Registering TS with master...
I20260812 19:11:43.344241 20664 heartbeater.cc:511] Master 127.19.70.190:45491 requested a full tablet report, sending...
I20260812 19:11:43.344274 19738 webserver.cc:460] Webserver started at http://127.19.70.131:39515/ using document root <none> and password file <none>
I20260812 19:11:43.344929 19738 fs_manager.cc:359] Metadata directory not provided
I20260812 19:11:43.345176 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:11:43.345517 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:11:43.346688 20423 ts_manager.cc:194] Registered new tserver with Master: c9645294d712440283be14a35f4e4beb (127.19.70.130:39101)
I20260812 19:11:43.347188 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/ts-2-root/instance:
uuid: "3e47e633f9784af7b18e39250f7211df"
format_stamp: "Formatted at 2026-08-12 19:11:43 on dist-test-slave-pgkr"
I20260812 19:11:43.349032 20423 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38878
I20260812 19:11:43.352972 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260812 19:11:43.356899 20678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:43.357816 19738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:11:43.358107 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/ts-2-root
uuid: "3e47e633f9784af7b18e39250f7211df"
format_stamp: "Formatted at 2026-08-12 19:11:43 on dist-test-slave-pgkr"
I20260812 19:11:43.358363 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786561868098184-19738-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:11:43.371429 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 19:11:43.372695 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:11:43.374147 19738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 19:11:43.376614 19738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:11:43.376832 19738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:43.377161 19738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 19:11:43.377318 19738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:11:43.420957 19738 rpc_server.cc:307] RPC server started. Bound to: 127.19.70.131:46591
I20260812 19:11:43.421070 20756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.70.131:46591 every 8 connection(s)
I20260812 19:11:43.437642 20757 heartbeater.cc:347] Connected to a master server at 127.19.70.190:45491
I20260812 19:11:43.438076 20757 heartbeater.cc:464] Registering TS with master...
I20260812 19:11:43.438866 20757 heartbeater.cc:511] Master 127.19.70.190:45491 requested a full tablet report, sending...
I20260812 19:11:43.441042 20423 ts_manager.cc:194] Registered new tserver with Master: 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:43.441808 19738 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017485122s
I20260812 19:11:43.443344 20423 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38880
I20260812 19:11:43.445017 19738 test_util.cc:268] Using random seed: 1537197052
I20260812 19:11:43.472500 20423 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38894:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 19:11:43.475569 20423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 19:11:43.583865 20623 tablet_service.cc:1467] Processing CreateTablet for tablet e3016afd35dd4d4288c4d55f403c268e (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260812 19:11:43.584831 20616 tablet_service.cc:1467] Processing CreateTablet for tablet 02bf63ed4d474c0a9b03bde1a2aa2f7a (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260812 19:11:43.585543 20623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3016afd35dd4d4288c4d55f403c268e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.586329 20616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bf63ed4d474c0a9b03bde1a2aa2f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.586282 20536 tablet_service.cc:1467] Processing CreateTablet for tablet 147c4ad62d7b4714815454a49c2abd7b (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260812 19:11:43.587478 20541 tablet_service.cc:1467] Processing CreateTablet for tablet 66f8b8ae72814bfbbf087d4847f8cde7 (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260812 19:11:43.587924 20536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147c4ad62d7b4714815454a49c2abd7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.587838 20622 tablet_service.cc:1467] Processing CreateTablet for tablet b1ecbe2f6bc54a53b637427ea3ba3fed (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260812 19:11:43.588858 20541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f8b8ae72814bfbbf087d4847f8cde7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.589311 20622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1ecbe2f6bc54a53b637427ea3ba3fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.592039 20621 tablet_service.cc:1467] Processing CreateTablet for tablet 6e56a1b8bf6042cc9ac59c1a4bcfa130 (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260812 19:11:43.593755 20621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e56a1b8bf6042cc9ac59c1a4bcfa130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.593461 20538 tablet_service.cc:1467] Processing CreateTablet for tablet b1ecbe2f6bc54a53b637427ea3ba3fed (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260812 19:11:43.595191 20710 tablet_service.cc:1467] Processing CreateTablet for tablet 66f8b8ae72814bfbbf087d4847f8cde7 (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260812 19:11:43.595901 20537 tablet_service.cc:1467] Processing CreateTablet for tablet 6e56a1b8bf6042cc9ac59c1a4bcfa130 (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260812 19:11:43.596788 20710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f8b8ae72814bfbbf087d4847f8cde7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.597357 20537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e56a1b8bf6042cc9ac59c1a4bcfa130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.597221 20619 tablet_service.cc:1467] Processing CreateTablet for tablet 3856d6355b564cb69521bca5f18c7599 (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260812 19:11:43.599957 20618 tablet_service.cc:1467] Processing CreateTablet for tablet e0edf0240b0a483cbf2315307b984365 (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260812 19:11:43.600196 20534 tablet_service.cc:1467] Processing CreateTablet for tablet 3856d6355b564cb69521bca5f18c7599 (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260812 19:11:43.601253 20708 tablet_service.cc:1467] Processing CreateTablet for tablet b1ecbe2f6bc54a53b637427ea3ba3fed (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260812 19:11:43.602583 20533 tablet_service.cc:1467] Processing CreateTablet for tablet e0edf0240b0a483cbf2315307b984365 (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260812 19:11:43.595930 20620 tablet_service.cc:1467] Processing CreateTablet for tablet 147c4ad62d7b4714815454a49c2abd7b (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260812 19:11:43.603866 20707 tablet_service.cc:1467] Processing CreateTablet for tablet 6e56a1b8bf6042cc9ac59c1a4bcfa130 (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260812 19:11:43.599961 20709 tablet_service.cc:1467] Processing CreateTablet for tablet e3016afd35dd4d4288c4d55f403c268e (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260812 19:11:43.605352 20532 tablet_service.cc:1467] Processing CreateTablet for tablet 02bf63ed4d474c0a9b03bde1a2aa2f7a (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260812 19:11:43.592042 20540 tablet_service.cc:1467] Processing CreateTablet for tablet e3016afd35dd4d4288c4d55f403c268e (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260812 19:11:43.606329 20706 tablet_service.cc:1467] Processing CreateTablet for tablet 147c4ad62d7b4714815454a49c2abd7b (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260812 19:11:43.602708 20624 tablet_service.cc:1467] Processing CreateTablet for tablet 66f8b8ae72814bfbbf087d4847f8cde7 (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260812 19:11:43.607975 20619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3856d6355b564cb69521bca5f18c7599. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.608781 20704 tablet_service.cc:1467] Processing CreateTablet for tablet 3856d6355b564cb69521bca5f18c7599 (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260812 19:11:43.609202 20530 tablet_service.cc:1467] Processing CreateTablet for tablet 0066afe62a194830a36fd1a214ea579a (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260812 19:11:43.611105 20624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f8b8ae72814bfbbf087d4847f8cde7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.611413 20703 tablet_service.cc:1467] Processing CreateTablet for tablet e0edf0240b0a483cbf2315307b984365 (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260812 19:11:43.614512 20702 tablet_service.cc:1467] Processing CreateTablet for tablet 02bf63ed4d474c0a9b03bde1a2aa2f7a (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260812 19:11:43.615166 20532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bf63ed4d474c0a9b03bde1a2aa2f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.617641 20534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3856d6355b564cb69521bca5f18c7599. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.618377 20620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147c4ad62d7b4714815454a49c2abd7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.617807 20701 tablet_service.cc:1467] Processing CreateTablet for tablet 0066afe62a194830a36fd1a214ea579a (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260812 19:11:43.621843 20529 tablet_service.cc:1467] Processing CreateTablet for tablet 40269a4340dd4476beeface532a7d1d0 (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260812 19:11:43.624511 20700 tablet_service.cc:1467] Processing CreateTablet for tablet 40269a4340dd4476beeface532a7d1d0 (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260812 19:11:43.626781 20708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1ecbe2f6bc54a53b637427ea3ba3fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.627679 20709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3016afd35dd4d4288c4d55f403c268e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.628770 20700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40269a4340dd4476beeface532a7d1d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.631174 20703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0edf0240b0a483cbf2315307b984365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.633970 20701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0066afe62a194830a36fd1a214ea579a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.635104 20704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3856d6355b564cb69521bca5f18c7599. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.636101 20615 tablet_service.cc:1467] Processing CreateTablet for tablet 0066afe62a194830a36fd1a214ea579a (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260812 19:11:43.636790 20707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e56a1b8bf6042cc9ac59c1a4bcfa130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.638747 20614 tablet_service.cc:1467] Processing CreateTablet for tablet 40269a4340dd4476beeface532a7d1d0 (DEFAULT_TABLE table=test-workload [id=5b529c6491554379890135e904f8dcf8]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260812 19:11:43.640383 20706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147c4ad62d7b4714815454a49c2abd7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.627746 20533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0edf0240b0a483cbf2315307b984365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.643028 20702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bf63ed4d474c0a9b03bde1a2aa2f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.647181 20540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3016afd35dd4d4288c4d55f403c268e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.653049 20618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0edf0240b0a483cbf2315307b984365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.653939 20614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40269a4340dd4476beeface532a7d1d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.654521 20529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40269a4340dd4476beeface532a7d1d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.655104 20615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0066afe62a194830a36fd1a214ea579a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.657167 20530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0066afe62a194830a36fd1a214ea579a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.658207 20538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1ecbe2f6bc54a53b637427ea3ba3fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:11:43.751963 20776 tablet_bootstrap.cc:492] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6: Bootstrap starting.
I20260812 19:11:43.767638 20776 tablet_bootstrap.cc:654] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:43.780560 20776 tablet_bootstrap.cc:492] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6: No bootstrap required, opened a new log
I20260812 19:11:43.782544 20776 ts_tablet_manager.cc:1397] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent bootstrapping tablet: real 0.035s user 0.014s sys 0.002s
I20260812 19:11:43.786309 20776 raft_consensus.cc:348] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:43.788826 20776 raft_consensus.cc:374] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:43.789252 20776 raft_consensus.cc:729] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Initialized, Role: FOLLOWER
I20260812 19:11:43.787882 20777 tablet_bootstrap.cc:492] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df: Bootstrap starting.
I20260812 19:11:43.790181 20776 consensus_queue.cc:260] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:43.840293 20777 tablet_bootstrap.cc:654] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:43.866537 20781 tablet_bootstrap.cc:492] T 66f8b8ae72814bfbbf087d4847f8cde7 P c9645294d712440283be14a35f4e4beb: Bootstrap starting.
I20260812 19:11:43.869747 20776 ts_tablet_manager.cc:1428] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent starting tablet: real 0.086s user 0.041s sys 0.030s
I20260812 19:11:43.873852 20579 heartbeater.cc:503] Master 127.19.70.190:45491 was elected leader, sending a full tablet report...
I20260812 19:11:43.880846 20777 tablet_bootstrap.cc:492] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df: No bootstrap required, opened a new log
I20260812 19:11:43.881397 20777 ts_tablet_manager.cc:1397] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df: Time spent bootstrapping tablet: real 0.094s user 0.014s sys 0.018s
I20260812 19:11:43.881873 20776 tablet_bootstrap.cc:492] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6: Bootstrap starting.
I20260812 19:11:43.882689 20781 tablet_bootstrap.cc:654] T 66f8b8ae72814bfbbf087d4847f8cde7 P c9645294d712440283be14a35f4e4beb: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:43.884562 20777 raft_consensus.cc:348] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:43.885455 20777 raft_consensus.cc:374] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:43.885771 20777 raft_consensus.cc:729] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Initialized, Role: FOLLOWER
I20260812 19:11:43.886749 20777 consensus_queue.cc:260] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:43.889693 20776 tablet_bootstrap.cc:654] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:43.895097 20777 ts_tablet_manager.cc:1428] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260812 19:11:43.897241 20757 heartbeater.cc:503] Master 127.19.70.190:45491 was elected leader, sending a full tablet report...
I20260812 19:11:43.899394 20781 tablet_bootstrap.cc:492] T 66f8b8ae72814bfbbf087d4847f8cde7 P c9645294d712440283be14a35f4e4beb: No bootstrap required, opened a new log
I20260812 19:11:43.901482 20777 tablet_bootstrap.cc:492] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df: Bootstrap starting.
I20260812 19:11:43.903048 20781 ts_tablet_manager.cc:1397] T 66f8b8ae72814bfbbf087d4847f8cde7 P c9645294d712440283be14a35f4e4beb: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.013s
I20260812 19:11:43.905766 20781 raft_consensus.cc:348] T 66f8b8ae72814bfbbf087d4847f8cde7 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:43.906472 20781 raft_consensus.cc:374] T 66f8b8ae72814bfbbf087d4847f8cde7 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:43.906793 20781 raft_consensus.cc:729] T 66f8b8ae72814bfbbf087d4847f8cde7 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9645294d712440283be14a35f4e4beb, State: Initialized, Role: FOLLOWER
I20260812 19:11:43.907554 20781 consensus_queue.cc:260] T 66f8b8ae72814bfbbf087d4847f8cde7 P c9645294d712440283be14a35f4e4beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:43.909658 20777 tablet_bootstrap.cc:654] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:43.910027 20664 heartbeater.cc:503] Master 127.19.70.190:45491 was elected leader, sending a full tablet report...
I20260812 19:11:43.912837 20776 tablet_bootstrap.cc:492] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6: No bootstrap required, opened a new log
I20260812 19:11:43.913416 20776 ts_tablet_manager.cc:1397] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.012s
I20260812 19:11:43.916487 20776 raft_consensus.cc:348] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:43.917544 20776 raft_consensus.cc:374] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:43.918031 20776 raft_consensus.cc:729] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Initialized, Role: FOLLOWER
I20260812 19:11:43.919377 20781 ts_tablet_manager.cc:1428] T 66f8b8ae72814bfbbf087d4847f8cde7 P c9645294d712440283be14a35f4e4beb: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260812 19:11:43.919068 20776 consensus_queue.cc:260] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:43.922067 20777 tablet_bootstrap.cc:492] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df: No bootstrap required, opened a new log
I20260812 19:11:43.922255 20776 ts_tablet_manager.cc:1428] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260812 19:11:43.922794 20777 ts_tablet_manager.cc:1397] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.012s
I20260812 19:11:43.923512 20780 raft_consensus.cc:482] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:43.924032 20780 raft_consensus.cc:504] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:43.924923 20781 tablet_bootstrap.cc:492] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb: Bootstrap starting.
I20260812 19:11:43.925848 20777 raft_consensus.cc:348] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
W20260812 19:11:43.927094 20758 tablet.cc:2369] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 19:11:43.930876 20780 leader_election.cc:288] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:43.926699 20777 raft_consensus.cc:374] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:43.941833 20776 tablet_bootstrap.cc:492] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6: Bootstrap starting.
I20260812 19:11:43.943276 20780 raft_consensus.cc:482] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:43.943846 20780 raft_consensus.cc:504] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:43.938403 20777 raft_consensus.cc:729] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Initialized, Role: FOLLOWER
I20260812 19:11:43.944977 20781 tablet_bootstrap.cc:654] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:43.946707 20780 leader_election.cc:288] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:43.949802 20776 tablet_bootstrap.cc:654] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:43.950326 20777 consensus_queue.cc:260] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:43.956337 20777 ts_tablet_manager.cc:1428] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20260812 19:11:43.957263 20777 tablet_bootstrap.cc:492] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df: Bootstrap starting.
I20260812 19:11:43.966346 20777 tablet_bootstrap.cc:654] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:43.969341 20776 tablet_bootstrap.cc:492] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6: No bootstrap required, opened a new log
I20260812 19:11:43.969892 20776 ts_tablet_manager.cc:1397] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.013s
I20260812 19:11:43.972849 20776 raft_consensus.cc:348] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:43.973694 20776 raft_consensus.cc:374] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:43.974021 20776 raft_consensus.cc:729] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Initialized, Role: FOLLOWER
I20260812 19:11:43.975006 20776 consensus_queue.cc:260] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:43.976861 20781 tablet_bootstrap.cc:492] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb: No bootstrap required, opened a new log
I20260812 19:11:43.977442 20781 ts_tablet_manager.cc:1397] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb: Time spent bootstrapping tablet: real 0.057s user 0.013s sys 0.008s
I20260812 19:11:43.979417 20776 ts_tablet_manager.cc:1428] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260812 19:11:43.980494 20776 tablet_bootstrap.cc:492] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6: Bootstrap starting.
I20260812 19:11:43.980194 20781 raft_consensus.cc:348] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:43.980917 20781 raft_consensus.cc:374] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:43.981221 20781 raft_consensus.cc:729] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9645294d712440283be14a35f4e4beb, State: Initialized, Role: FOLLOWER
I20260812 19:11:43.981926 20781 consensus_queue.cc:260] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:43.986367 20781 ts_tablet_manager.cc:1428] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260812 19:11:43.987212 20776 tablet_bootstrap.cc:654] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:43.987509 20781 tablet_bootstrap.cc:492] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb: Bootstrap starting.
I20260812 19:11:43.990940 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0066afe62a194830a36fd1a214ea579a" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb" is_pre_election: true
I20260812 19:11:43.993433 20635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147c4ad62d7b4714815454a49c2abd7b" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb" is_pre_election: true
I20260812 19:11:43.996131 20777 tablet_bootstrap.cc:492] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df: No bootstrap required, opened a new log
I20260812 19:11:43.996728 20777 ts_tablet_manager.cc:1397] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df: Time spent bootstrapping tablet: real 0.040s user 0.014s sys 0.003s
I20260812 19:11:43.999760 20781 tablet_bootstrap.cc:654] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:43.999373 20777 raft_consensus.cc:348] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
W20260812 19:11:43.995838 20510 leader_election.cc:341] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9645294d712440283be14a35f4e4beb (127.19.70.130:39101): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.000281 20777 raft_consensus.cc:374] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.000635 20777 raft_consensus.cc:729] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Initialized, Role: FOLLOWER
W20260812 19:11:44.000983 20510 leader_election.cc:341] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9645294d712440283be14a35f4e4beb (127.19.70.130:39101): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.001662 20777 consensus_queue.cc:260] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.016345 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0066afe62a194830a36fd1a214ea579a" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df" is_pre_election: true
I20260812 19:11:44.017306 20723 raft_consensus.cc:2454] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 0.
I20260812 19:11:44.017742 20777 ts_tablet_manager.cc:1428] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df: Time spent starting tablet: real 0.021s user 0.009s sys 0.000s
W20260812 19:11:44.018954 20488 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260812 19:11:44.018733 20722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147c4ad62d7b4714815454a49c2abd7b" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df" is_pre_election: true
I20260812 19:11:44.019644 20722 raft_consensus.cc:2454] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 0.
I20260812 19:11:44.020846 20512 leader_election.cc:302] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df; no voters: c9645294d712440283be14a35f4e4beb
I20260812 19:11:44.021804 20780 raft_consensus.cc:2781] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:44.022158 20780 raft_consensus.cc:482] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:44.022552 20780 raft_consensus.cc:3037] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.022332 20512 leader_election.cc:302] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df; no voters: c9645294d712440283be14a35f4e4beb
I20260812 19:11:44.030388 20777 tablet_bootstrap.cc:492] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df: Bootstrap starting.
I20260812 19:11:44.033828 20780 raft_consensus.cc:504] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.035773 20795 raft_consensus.cc:2781] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:44.037242 20635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0066afe62a194830a36fd1a214ea579a" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:44.038005 20776 tablet_bootstrap.cc:492] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6: No bootstrap required, opened a new log
I20260812 19:11:44.038509 20776 ts_tablet_manager.cc:1397] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent bootstrapping tablet: real 0.058s user 0.021s sys 0.011s
W20260812 19:11:44.038944 20510 leader_election.cc:341] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9645294d712440283be14a35f4e4beb (127.19.70.130:39101): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.040100 20795 raft_consensus.cc:482] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:44.042037 20781 tablet_bootstrap.cc:492] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb: No bootstrap required, opened a new log
I20260812 19:11:44.041962 20777 tablet_bootstrap.cc:654] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.042651 20795 raft_consensus.cc:3037] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.042091 20776 raft_consensus.cc:348] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.043222 20776 raft_consensus.cc:374] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.043576 20776 raft_consensus.cc:729] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.044977 20780 leader_election.cc:288] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Requested vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.045192 20722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0066afe62a194830a36fd1a214ea579a" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:11:44.045866 20722 raft_consensus.cc:3037] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.046098 20781 ts_tablet_manager.cc:1397] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb: Time spent bootstrapping tablet: real 0.059s user 0.029s sys 0.020s
I20260812 19:11:44.049242 20777 tablet_bootstrap.cc:492] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df: No bootstrap required, opened a new log
I20260812 19:11:44.049853 20777 ts_tablet_manager.cc:1397] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.003s
I20260812 19:11:44.049732 20781 raft_consensus.cc:348] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.050805 20781 raft_consensus.cc:374] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.051203 20781 raft_consensus.cc:729] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9645294d712440283be14a35f4e4beb, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.044466 20776 consensus_queue.cc:260] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.052088 20781 consensus_queue.cc:260] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.054420 20722 raft_consensus.cc:2454] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 1.
I20260812 19:11:44.054842 20781 ts_tablet_manager.cc:1428] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260812 19:11:44.055861 20512 leader_election.cc:302] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df; no voters: c9645294d712440283be14a35f4e4beb
I20260812 19:11:44.056222 20781 tablet_bootstrap.cc:492] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb: Bootstrap starting.
I20260812 19:11:44.057199 20780 raft_consensus.cc:2781] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:44.061009 20776 ts_tablet_manager.cc:1428] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent starting tablet: real 0.022s user 0.007s sys 0.001s
I20260812 19:11:44.062186 20776 tablet_bootstrap.cc:492] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6: Bootstrap starting.
I20260812 19:11:44.065048 20781 tablet_bootstrap.cc:654] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.066699 20777 raft_consensus.cc:348] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.067588 20777 raft_consensus.cc:374] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.067917 20777 raft_consensus.cc:729] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.068717 20777 consensus_queue.cc:260] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.071456 20777 ts_tablet_manager.cc:1428] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df: Time spent starting tablet: real 0.021s user 0.001s sys 0.008s
I20260812 19:11:44.072607 20777 tablet_bootstrap.cc:492] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df: Bootstrap starting.
I20260812 19:11:44.074920 20795 raft_consensus.cc:504] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.079427 20795 leader_election.cc:288] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Requested vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.079905 20635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147c4ad62d7b4714815454a49c2abd7b" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:44.081157 20780 raft_consensus.cc:686] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Becoming Leader. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
W20260812 19:11:44.081699 20510 leader_election.cc:341] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9645294d712440283be14a35f4e4beb (127.19.70.130:39101): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.082355 20780 consensus_queue.cc:237] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.083967 20776 tablet_bootstrap.cc:654] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.080163 20795 raft_consensus.cc:482] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:44.087116 20722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147c4ad62d7b4714815454a49c2abd7b" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:11:44.087800 20722 raft_consensus.cc:3037] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.090744 20795 raft_consensus.cc:504] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.093111 20795 leader_election.cc:288] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.095453 20635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f8b8ae72814bfbbf087d4847f8cde7" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb" is_pre_election: true
I20260812 19:11:44.096614 20635 raft_consensus.cc:2454] T 66f8b8ae72814bfbbf087d4847f8cde7 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 0.
I20260812 19:11:44.098563 20510 leader_election.cc:302] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:11:44.100858 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f8b8ae72814bfbbf087d4847f8cde7" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df" is_pre_election: true
I20260812 19:11:44.102886 20795 raft_consensus.cc:2781] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260812 19:11:44.103117 20512 leader_election.cc:341] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.107797 20795 raft_consensus.cc:482] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:44.111531 20776 tablet_bootstrap.cc:492] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6: No bootstrap required, opened a new log
I20260812 19:11:44.112098 20776 ts_tablet_manager.cc:1397] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent bootstrapping tablet: real 0.050s user 0.013s sys 0.009s
I20260812 19:11:44.112453 20722 raft_consensus.cc:2454] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 1.
I20260812 19:11:44.113813 20777 tablet_bootstrap.cc:654] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.115387 20776 raft_consensus.cc:348] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.114703 20423 catalog_manager.cc:5495] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:44.116282 20776 raft_consensus.cc:374] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.116667 20776 raft_consensus.cc:729] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.108453 20795 raft_consensus.cc:3037] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.117631 20776 consensus_queue.cc:260] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.118091 20512 leader_election.cc:302] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df; no voters: c9645294d712440283be14a35f4e4beb
I20260812 19:11:44.119402 20798 raft_consensus.cc:2781] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:44.120157 20798 raft_consensus.cc:686] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Becoming Leader. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:44.120352 20776 ts_tablet_manager.cc:1428] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 19:11:44.121383 20781 tablet_bootstrap.cc:492] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb: No bootstrap required, opened a new log
I20260812 19:11:44.121392 20776 tablet_bootstrap.cc:492] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6: Bootstrap starting.
I20260812 19:11:44.121073 20798 consensus_queue.cc:237] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.122228 20781 ts_tablet_manager.cc:1397] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb: Time spent bootstrapping tablet: real 0.066s user 0.043s sys 0.015s
I20260812 19:11:44.125319 20781 raft_consensus.cc:348] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.126266 20781 raft_consensus.cc:374] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.126662 20781 raft_consensus.cc:729] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9645294d712440283be14a35f4e4beb, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.127526 20781 consensus_queue.cc:260] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.129245 20776 tablet_bootstrap.cc:654] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.128863 20795 raft_consensus.cc:504] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.132200 20635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f8b8ae72814bfbbf087d4847f8cde7" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:44.132905 20635 raft_consensus.cc:3037] T 66f8b8ae72814bfbbf087d4847f8cde7 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.133041 20777 tablet_bootstrap.cc:492] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df: No bootstrap required, opened a new log
I20260812 19:11:44.133761 20777 ts_tablet_manager.cc:1397] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.005s
I20260812 19:11:44.137748 20777 raft_consensus.cc:348] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.140010 20777 raft_consensus.cc:374] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.140354 20777 raft_consensus.cc:729] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.141108 20777 consensus_queue.cc:260] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.141664 20781 ts_tablet_manager.cc:1428] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260812 19:11:44.142390 20635 raft_consensus.cc:2454] T 66f8b8ae72814bfbbf087d4847f8cde7 P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 1.
I20260812 19:11:44.143545 20777 ts_tablet_manager.cc:1428] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260812 19:11:44.143797 20510 leader_election.cc:302] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:11:44.144819 20798 raft_consensus.cc:2781] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:44.145385 20798 raft_consensus.cc:686] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Becoming Leader. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:44.145642 20795 leader_election.cc:288] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Requested vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.146811 20781 tablet_bootstrap.cc:492] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P c9645294d712440283be14a35f4e4beb: Bootstrap starting.
I20260812 19:11:44.146317 20798 consensus_queue.cc:237] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.146992 20722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f8b8ae72814bfbbf087d4847f8cde7" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df"
W20260812 19:11:44.148689 20512 leader_election.cc:341] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.144795 20777 tablet_bootstrap.cc:492] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 3e47e633f9784af7b18e39250f7211df: Bootstrap starting.
I20260812 19:11:44.156955 20784 raft_consensus.cc:482] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:44.157678 20784 raft_consensus.cc:504] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.158488 20776 tablet_bootstrap.cc:492] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6: No bootstrap required, opened a new log
I20260812 19:11:44.158787 20777 tablet_bootstrap.cc:654] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 3e47e633f9784af7b18e39250f7211df: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.159169 20776 ts_tablet_manager.cc:1397] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.012s
I20260812 19:11:44.157346 20781 tablet_bootstrap.cc:654] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P c9645294d712440283be14a35f4e4beb: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.181344 20784 leader_election.cc:288] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), c9645294d712440283be14a35f4e4beb (127.19.70.130:39101)
I20260812 19:11:44.185104 20776 raft_consensus.cc:348] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.186357 20776 raft_consensus.cc:374] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.186712 20798 raft_consensus.cc:482] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:44.186764 20776 raft_consensus.cc:729] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.187320 20798 raft_consensus.cc:504] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.187783 20776 consensus_queue.cc:260] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.189580 20798 leader_election.cc:288] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.190584 20776 ts_tablet_manager.cc:1428] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent starting tablet: real 0.031s user 0.007s sys 0.001s
I20260812 19:11:44.191694 20776 tablet_bootstrap.cc:492] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6: Bootstrap starting.
I20260812 19:11:44.191440 20635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bf63ed4d474c0a9b03bde1a2aa2f7a" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb" is_pre_election: true
I20260812 19:11:44.192330 20635 raft_consensus.cc:2454] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 0.
I20260812 19:11:44.194036 20510 leader_election.cc:302] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:11:44.195063 20798 raft_consensus.cc:2781] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:44.195451 20798 raft_consensus.cc:482] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:44.195806 20798 raft_consensus.cc:3037] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.200644 20776 tablet_bootstrap.cc:654] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.203892 20798 raft_consensus.cc:504] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.207862 20635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bf63ed4d474c0a9b03bde1a2aa2f7a" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:44.208817 20635 raft_consensus.cc:3037] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.189772 20423 catalog_manager.cc:5495] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:44.212335 20423 catalog_manager.cc:5495] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:44.213899 20780 raft_consensus.cc:482] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:44.214459 20780 raft_consensus.cc:504] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.216737 20780 leader_election.cc:288] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.218279 20635 raft_consensus.cc:2454] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 1.
I20260812 19:11:44.218649 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40269a4340dd4476beeface532a7d1d0" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb" is_pre_election: true
I20260812 19:11:44.219400 20636 raft_consensus.cc:2454] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 0.
I20260812 19:11:44.219553 20510 leader_election.cc:302] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:11:44.220534 20780 raft_consensus.cc:2781] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:44.220583 20510 leader_election.cc:302] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:11:44.221470 20795 raft_consensus.cc:2781] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:44.221278 20722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bf63ed4d474c0a9b03bde1a2aa2f7a" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df" is_pre_election: true
I20260812 19:11:44.221906 20795 raft_consensus.cc:482] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:44.223048 20795 raft_consensus.cc:3037] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.221299 20780 raft_consensus.cc:686] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Becoming Leader. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:44.222251 20798 leader_election.cc:288] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Requested vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.224583 20777 tablet_bootstrap.cc:492] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 3e47e633f9784af7b18e39250f7211df: No bootstrap required, opened a new log
I20260812 19:11:44.224448 20780 consensus_queue.cc:237] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.225411 20777 ts_tablet_manager.cc:1397] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 3e47e633f9784af7b18e39250f7211df: Time spent bootstrapping tablet: real 0.081s user 0.023s sys 0.015s
I20260812 19:11:44.228296 20722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40269a4340dd4476beeface532a7d1d0" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df" is_pre_election: true
I20260812 19:11:44.228418 20777 raft_consensus.cc:348] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.229364 20777 raft_consensus.cc:374] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.229691 20777 raft_consensus.cc:729] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.230418 20777 consensus_queue.cc:260] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 3e47e633f9784af7b18e39250f7211df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.230952 20795 raft_consensus.cc:504] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.231328 20722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bf63ed4d474c0a9b03bde1a2aa2f7a" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df"
W20260812 19:11:44.233333 20512 leader_election.cc:341] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.234051 20795 leader_election.cc:288] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Requested vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.234156 20635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40269a4340dd4476beeface532a7d1d0" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:44.241680 20635 raft_consensus.cc:3037] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Advancing to term 1
W20260812 19:11:44.242091 20512 leader_election.cc:341] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591): Illegal state: must be running to vote when last-logged opid is not known
W20260812 19:11:44.243140 20512 leader_election.cc:341] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.245277 20722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40269a4340dd4476beeface532a7d1d0" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df"
W20260812 19:11:44.246979 20512 leader_election.cc:341] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.249627 20635 raft_consensus.cc:2454] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 1.
I20260812 19:11:44.250926 20510 leader_election.cc:302] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, c9645294d712440283be14a35f4e4beb; no voters: 3e47e633f9784af7b18e39250f7211df
I20260812 19:11:44.252210 20795 raft_consensus.cc:2781] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:44.252779 20795 raft_consensus.cc:686] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Becoming Leader. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:44.253741 20795 consensus_queue.cc:237] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.236748 20777 ts_tablet_manager.cc:1428] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 3e47e633f9784af7b18e39250f7211df: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260812 19:11:44.258708 20777 tablet_bootstrap.cc:492] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df: Bootstrap starting.
I20260812 19:11:44.263441 20781 tablet_bootstrap.cc:492] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P c9645294d712440283be14a35f4e4beb: No bootstrap required, opened a new log
I20260812 19:11:44.264178 20781 ts_tablet_manager.cc:1397] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P c9645294d712440283be14a35f4e4beb: Time spent bootstrapping tablet: real 0.121s user 0.025s sys 0.057s
I20260812 19:11:44.265229 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" is_pre_election: true
I20260812 19:11:44.266131 20777 tablet_bootstrap.cc:654] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df: Neither blocks nor log segments found. Creating new log.
W20260812 19:11:44.267342 20681 leader_election.cc:341] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.267377 20781 raft_consensus.cc:348] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.268312 20781 raft_consensus.cc:374] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.268723 20781 raft_consensus.cc:729] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9645294d712440283be14a35f4e4beb, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.269508 20781 consensus_queue.cc:260] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P c9645294d712440283be14a35f4e4beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.272055 20781 ts_tablet_manager.cc:1428] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P c9645294d712440283be14a35f4e4beb: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260812 19:11:44.273180 20781 tablet_bootstrap.cc:492] T 147c4ad62d7b4714815454a49c2abd7b P c9645294d712440283be14a35f4e4beb: Bootstrap starting.
I20260812 19:11:44.271754 20421 catalog_manager.cc:5495] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:44.276633 20786 raft_consensus.cc:482] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:44.277235 20786 raft_consensus.cc:504] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.278024 20776 tablet_bootstrap.cc:492] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6: No bootstrap required, opened a new log
I20260812 19:11:44.280051 20781 tablet_bootstrap.cc:654] T 147c4ad62d7b4714815454a49c2abd7b P c9645294d712440283be14a35f4e4beb: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.282693 20784 raft_consensus.cc:482] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:44.282966 20786 leader_election.cc:288] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.283419 20784 raft_consensus.cc:504] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.286691 20784 leader_election.cc:288] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), c9645294d712440283be14a35f4e4beb (127.19.70.130:39101)
I20260812 19:11:44.275014 20421 catalog_manager.cc:5495] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:44.287624 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" is_pre_election: true
W20260812 19:11:44.290023 20681 leader_election.cc:341] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.278740 20776 ts_tablet_manager.cc:1397] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent bootstrapping tablet: real 0.087s user 0.017s sys 0.013s
I20260812 19:11:44.301285 20776 raft_consensus.cc:348] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.302538 20776 raft_consensus.cc:374] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.302983 20776 raft_consensus.cc:729] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.303962 20776 consensus_queue.cc:260] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.308069 20776 ts_tablet_manager.cc:1428] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent starting tablet: real 0.017s user 0.007s sys 0.002s
I20260812 19:11:44.308905 20776 tablet_bootstrap.cc:492] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6: Bootstrap starting.
I20260812 19:11:44.315043 20776 tablet_bootstrap.cc:654] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.320531 20777 tablet_bootstrap.cc:492] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df: No bootstrap required, opened a new log
I20260812 19:11:44.321085 20777 ts_tablet_manager.cc:1397] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df: Time spent bootstrapping tablet: real 0.063s user 0.024s sys 0.024s
I20260812 19:11:44.323098 20795 raft_consensus.cc:482] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:44.323670 20795 raft_consensus.cc:504] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.328037 20635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb" is_pre_election: true
I20260812 19:11:44.328450 20634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e56a1b8bf6042cc9ac59c1a4bcfa130" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb" is_pre_election: true
I20260812 19:11:44.328341 20777 raft_consensus.cc:348] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.329208 20634 raft_consensus.cc:2454] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 0.
I20260812 19:11:44.329486 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb" is_pre_election: true
I20260812 19:11:44.329294 20777 raft_consensus.cc:374] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.330368 20777 raft_consensus.cc:729] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.331329 20510 leader_election.cc:302] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:11:44.331245 20777 consensus_queue.cc:260] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
W20260812 19:11:44.332094 20680 leader_election.cc:341] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9645294d712440283be14a35f4e4beb (127.19.70.130:39101): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.332291 20780 raft_consensus.cc:2781] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:44.332659 20780 raft_consensus.cc:482] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:44.332618 20680 leader_election.cc:302] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e47e633f9784af7b18e39250f7211df; no voters: 2862c0a6eacd49d4a3982d8169e35fa6, c9645294d712440283be14a35f4e4beb
I20260812 19:11:44.333093 20780 raft_consensus.cc:3037] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.327036 20795 leader_election.cc:288] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.334992 20784 raft_consensus.cc:2726] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260812 19:11:44.334059 20680 leader_election.cc:341] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9645294d712440283be14a35f4e4beb (127.19.70.130:39101): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.336498 20680 leader_election.cc:302] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e47e633f9784af7b18e39250f7211df; no voters: 2862c0a6eacd49d4a3982d8169e35fa6, c9645294d712440283be14a35f4e4beb
I20260812 19:11:44.337417 20784 raft_consensus.cc:2726] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:11:44.336750 20777 ts_tablet_manager.cc:1428] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20260812 19:11:44.340768 20777 tablet_bootstrap.cc:492] T 66f8b8ae72814bfbbf087d4847f8cde7 P 3e47e633f9784af7b18e39250f7211df: Bootstrap starting.
I20260812 19:11:44.340868 20780 raft_consensus.cc:504] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.342933 20776 tablet_bootstrap.cc:492] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6: No bootstrap required, opened a new log
I20260812 19:11:44.343484 20776 ts_tablet_manager.cc:1397] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.003s
I20260812 19:11:44.344074 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e56a1b8bf6042cc9ac59c1a4bcfa130" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:44.344785 20636 raft_consensus.cc:3037] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.346487 20722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e56a1b8bf6042cc9ac59c1a4bcfa130" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df" is_pre_election: true
I20260812 19:11:44.347268 20722 raft_consensus.cc:2454] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 0.
I20260812 19:11:44.348452 20784 raft_consensus.cc:482] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:44.348971 20784 raft_consensus.cc:504] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.354220 20776 raft_consensus.cc:348] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.355422 20776 raft_consensus.cc:374] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.355784 20776 raft_consensus.cc:729] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.357385 20634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb" is_pre_election: true
I20260812 19:11:44.356858 20776 consensus_queue.cc:260] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.358172 20634 raft_consensus.cc:2454] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e47e633f9784af7b18e39250f7211df in term 0.
I20260812 19:11:44.357424 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" is_pre_election: true
I20260812 19:11:44.359344 20551 raft_consensus.cc:2454] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e47e633f9784af7b18e39250f7211df in term 0.
I20260812 19:11:44.362095 20681 leader_election.cc:302] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df; no voters:
I20260812 19:11:44.351184 20784 leader_election.cc:288] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), c9645294d712440283be14a35f4e4beb (127.19.70.130:39101)
I20260812 19:11:44.351230 20777 tablet_bootstrap.cc:654] T 66f8b8ae72814bfbbf087d4847f8cde7 P 3e47e633f9784af7b18e39250f7211df: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.369532 20776 ts_tablet_manager.cc:1428] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent starting tablet: real 0.026s user 0.005s sys 0.005s
I20260812 19:11:44.370990 20776 tablet_bootstrap.cc:492] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6: Bootstrap starting.
I20260812 19:11:44.369839 20784 raft_consensus.cc:2781] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:44.381826 20776 tablet_bootstrap.cc:654] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.374993 20784 raft_consensus.cc:482] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:44.381599 20722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e56a1b8bf6042cc9ac59c1a4bcfa130" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:11:44.383206 20784 raft_consensus.cc:3037] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.383461 20722 raft_consensus.cc:3037] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.375192 20780 leader_election.cc:288] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Requested vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.412917 20722 raft_consensus.cc:2454] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 1.
I20260812 19:11:44.414258 20512 leader_election.cc:302] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df; no voters:
I20260812 19:11:44.415231 20780 raft_consensus.cc:2781] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:44.415627 20784 raft_consensus.cc:504] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.416646 20780 raft_consensus.cc:686] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Becoming Leader. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:44.417550 20636 raft_consensus.cc:2454] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 1.
I20260812 19:11:44.417805 20780 consensus_queue.cc:237] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.419143 20795 raft_consensus.cc:482] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:44.419273 20784 leader_election.cc:288] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 1 election: Requested vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), c9645294d712440283be14a35f4e4beb (127.19.70.130:39101)
I20260812 19:11:44.419605 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
I20260812 19:11:44.419749 20795 raft_consensus.cc:504] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.420248 20551 raft_consensus.cc:3037] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.426163 20634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb" is_pre_election: true
I20260812 19:11:44.426786 20550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" is_pre_election: true
I20260812 19:11:44.426388 20795 leader_election.cc:288] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
W20260812 19:11:44.427897 20510 leader_election.cc:341] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9645294d712440283be14a35f4e4beb (127.19.70.130:39101): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.428859 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:44.429390 20636 raft_consensus.cc:3037] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.430470 20781 tablet_bootstrap.cc:492] T 147c4ad62d7b4714815454a49c2abd7b P c9645294d712440283be14a35f4e4beb: No bootstrap required, opened a new log
I20260812 19:11:44.431064 20781 ts_tablet_manager.cc:1397] T 147c4ad62d7b4714815454a49c2abd7b P c9645294d712440283be14a35f4e4beb: Time spent bootstrapping tablet: real 0.158s user 0.024s sys 0.066s
I20260812 19:11:44.434230 20781 raft_consensus.cc:348] T 147c4ad62d7b4714815454a49c2abd7b P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.434953 20636 raft_consensus.cc:2454] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e47e633f9784af7b18e39250f7211df in term 1.
I20260812 19:11:44.435163 20781 raft_consensus.cc:374] T 147c4ad62d7b4714815454a49c2abd7b P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.435582 20781 raft_consensus.cc:729] T 147c4ad62d7b4714815454a49c2abd7b P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9645294d712440283be14a35f4e4beb, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.436024 20551 raft_consensus.cc:2454] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e47e633f9784af7b18e39250f7211df in term 1.
I20260812 19:11:44.436230 20680 leader_election.cc:302] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e47e633f9784af7b18e39250f7211df, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:11:44.436584 20781 consensus_queue.cc:260] T 147c4ad62d7b4714815454a49c2abd7b P c9645294d712440283be14a35f4e4beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.438325 20776 tablet_bootstrap.cc:492] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6: No bootstrap required, opened a new log
I20260812 19:11:44.438841 20776 ts_tablet_manager.cc:1397] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent bootstrapping tablet: real 0.068s user 0.016s sys 0.000s
I20260812 19:11:44.439198 20781 ts_tablet_manager.cc:1428] T 147c4ad62d7b4714815454a49c2abd7b P c9645294d712440283be14a35f4e4beb: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20260812 19:11:44.440132 20781 tablet_bootstrap.cc:492] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb: Bootstrap starting.
I20260812 19:11:44.441593 20776 raft_consensus.cc:348] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.442365 20776 raft_consensus.cc:374] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.442956 20784 raft_consensus.cc:2781] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:44.443560 20776 raft_consensus.cc:729] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.444478 20784 raft_consensus.cc:686] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 1 LEADER]: Becoming Leader. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Running, Role: LEADER
I20260812 19:11:44.444978 20776 consensus_queue.cc:260] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.445572 20784 consensus_queue.cc:237] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.449150 20776 ts_tablet_manager.cc:1428] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260812 19:11:44.450189 20776 tablet_bootstrap.cc:492] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6: Bootstrap starting.
I20260812 19:11:44.451232 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df" is_pre_election: true
I20260812 19:11:44.453701 20777 tablet_bootstrap.cc:492] T 66f8b8ae72814bfbbf087d4847f8cde7 P 3e47e633f9784af7b18e39250f7211df: No bootstrap required, opened a new log
I20260812 19:11:44.453410 20595 leader_election.cc:302] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9645294d712440283be14a35f4e4beb; no voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df
I20260812 19:11:44.454452 20777 ts_tablet_manager.cc:1397] T 66f8b8ae72814bfbbf087d4847f8cde7 P 3e47e633f9784af7b18e39250f7211df: Time spent bootstrapping tablet: real 0.114s user 0.027s sys 0.013s
I20260812 19:11:44.454871 20786 raft_consensus.cc:2726] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:11:44.456000 20722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df" is_pre_election: true
I20260812 19:11:44.456763 20722 raft_consensus.cc:2454] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 0.
I20260812 19:11:44.457854 20777 raft_consensus.cc:348] T 66f8b8ae72814bfbbf087d4847f8cde7 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.458678 20777 raft_consensus.cc:374] T 66f8b8ae72814bfbbf087d4847f8cde7 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.458372 20512 leader_election.cc:302] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df; no voters: c9645294d712440283be14a35f4e4beb
I20260812 19:11:44.459033 20777 raft_consensus.cc:729] T 66f8b8ae72814bfbbf087d4847f8cde7 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.459578 20795 raft_consensus.cc:2781] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:44.460345 20795 raft_consensus.cc:482] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:44.459801 20777 consensus_queue.cc:260] T 66f8b8ae72814bfbbf087d4847f8cde7 P 3e47e633f9784af7b18e39250f7211df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.460728 20795 raft_consensus.cc:3037] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.462775 20781 tablet_bootstrap.cc:654] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.462926 20777 ts_tablet_manager.cc:1428] T 66f8b8ae72814bfbbf087d4847f8cde7 P 3e47e633f9784af7b18e39250f7211df: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 19:11:44.464304 20777 tablet_bootstrap.cc:492] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df: Bootstrap starting.
I20260812 19:11:44.469430 20776 tablet_bootstrap.cc:654] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.471545 20795 raft_consensus.cc:504] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.479831 20777 tablet_bootstrap.cc:654] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.482059 20795 leader_election.cc:288] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Requested vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.497325 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:44.498224 20781 tablet_bootstrap.cc:492] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb: No bootstrap required, opened a new log
I20260812 19:11:44.498885 20781 ts_tablet_manager.cc:1397] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.017s
I20260812 19:11:44.500366 20421 catalog_manager.cc:5495] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e47e633f9784af7b18e39250f7211df (127.19.70.131). New cstate: current_term: 1 leader_uuid: "3e47e633f9784af7b18e39250f7211df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: HEALTHY } } }
W20260812 19:11:44.502543 20510 leader_election.cc:341] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9645294d712440283be14a35f4e4beb (127.19.70.130:39101): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.501364 20781 raft_consensus.cc:348] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.506950 20781 raft_consensus.cc:374] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.507357 20781 raft_consensus.cc:729] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9645294d712440283be14a35f4e4beb, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.509106 20781 consensus_queue.cc:260] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.507320 20423 catalog_manager.cc:5495] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:44.511406 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:11:44.512080 20781 ts_tablet_manager.cc:1428] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb: Time spent starting tablet: real 0.013s user 0.006s sys 0.002s
I20260812 19:11:44.512153 20723 raft_consensus.cc:3037] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.513396 20781 tablet_bootstrap.cc:492] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb: Bootstrap starting.
W20260812 19:11:44.514947 20510 consensus_peers.cc:487] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 -> Peer c9645294d712440283be14a35f4e4beb (127.19.70.130:39101): Couldn't send request to peer c9645294d712440283be14a35f4e4beb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:11:44.523669 20781 tablet_bootstrap.cc:654] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.517808 20780 consensus_queue.cc:1035] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:11:44.534427 20777 tablet_bootstrap.cc:492] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df: No bootstrap required, opened a new log
I20260812 19:11:44.535920 20777 ts_tablet_manager.cc:1397] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df: Time spent bootstrapping tablet: real 0.072s user 0.021s sys 0.000s
I20260812 19:11:44.538708 20777 raft_consensus.cc:348] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.539525 20777 raft_consensus.cc:374] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.540171 20786 raft_consensus.cc:482] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:44.540659 20786 raft_consensus.cc:504] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.542830 20786 leader_election.cc:288] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.544480 20780 raft_consensus.cc:482] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:44.544787 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3016afd35dd4d4288c4d55f403c268e" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" is_pre_election: true
I20260812 19:11:44.545038 20780 raft_consensus.cc:504] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
W20260812 19:11:44.546854 20595 leader_election.cc:341] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.546797 20721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3016afd35dd4d4288c4d55f403c268e" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df" is_pre_election: true
I20260812 19:11:44.547359 20780 leader_election.cc:288] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.549005 20720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df" is_pre_election: true
I20260812 19:11:44.549201 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb" is_pre_election: true
I20260812 19:11:44.549796 20720 raft_consensus.cc:2454] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 0.
I20260812 19:11:44.550994 20512 leader_election.cc:302] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df; no voters:
W20260812 19:11:44.551643 20510 leader_election.cc:341] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9645294d712440283be14a35f4e4beb (127.19.70.130:39101): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.551887 20780 raft_consensus.cc:2781] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:44.552294 20780 raft_consensus.cc:482] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:44.552652 20780 raft_consensus.cc:3037] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.556962 20776 tablet_bootstrap.cc:492] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6: No bootstrap required, opened a new log
I20260812 19:11:44.557443 20776 ts_tablet_manager.cc:1397] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent bootstrapping tablet: real 0.107s user 0.016s sys 0.014s
I20260812 19:11:44.532214 20723 raft_consensus.cc:2454] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 1.
I20260812 19:11:44.559487 20512 leader_election.cc:302] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df; no voters: c9645294d712440283be14a35f4e4beb
I20260812 19:11:44.560607 20795 raft_consensus.cc:2781] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:44.561200 20795 raft_consensus.cc:686] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Becoming Leader. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:44.562417 20795 consensus_queue.cc:237] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.562884 20780 raft_consensus.cc:504] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.566099 20780 leader_election.cc:288] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Requested vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.566936 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:44.567605 20720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:11:44.568701 20720 raft_consensus.cc:3037] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.539798 20777 raft_consensus.cc:729] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Initialized, Role: FOLLOWER
W20260812 19:11:44.570926 20510 leader_election.cc:341] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9645294d712440283be14a35f4e4beb (127.19.70.130:39101): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:11:44.572860 20781 tablet_bootstrap.cc:492] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb: No bootstrap required, opened a new log
I20260812 19:11:44.573565 20781 ts_tablet_manager.cc:1397] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb: Time spent bootstrapping tablet: real 0.060s user 0.025s sys 0.020s
I20260812 19:11:44.573844 20776 raft_consensus.cc:348] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.574812 20776 raft_consensus.cc:374] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.575127 20776 raft_consensus.cc:729] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.571255 20777 consensus_queue.cc:260] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.576483 20781 raft_consensus.cc:348] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.586892 20781 raft_consensus.cc:374] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.587365 20781 raft_consensus.cc:729] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9645294d712440283be14a35f4e4beb, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.584672 20423 catalog_manager.cc:5495] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:44.576308 20721 raft_consensus.cc:2454] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9645294d712440283be14a35f4e4beb in term 0.
I20260812 19:11:44.588407 20781 consensus_queue.cc:260] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.591754 20595 leader_election.cc:302] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e47e633f9784af7b18e39250f7211df, c9645294d712440283be14a35f4e4beb; no voters: 2862c0a6eacd49d4a3982d8169e35fa6
I20260812 19:11:44.592913 20720 raft_consensus.cc:2454] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 1.
I20260812 19:11:44.594291 20512 leader_election.cc:302] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df; no voters: c9645294d712440283be14a35f4e4beb
I20260812 19:11:44.595302 20780 raft_consensus.cc:2781] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:44.595820 20780 raft_consensus.cc:686] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Becoming Leader. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:44.596737 20780 consensus_queue.cc:237] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.598335 20786 raft_consensus.cc:2781] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:11:44.598814 20786 raft_consensus.cc:482] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:11:44.599174 20786 raft_consensus.cc:3037] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.601141 20777 ts_tablet_manager.cc:1428] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df: Time spent starting tablet: real 0.065s user 0.001s sys 0.008s
I20260812 19:11:44.602131 20777 tablet_bootstrap.cc:492] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df: Bootstrap starting.
I20260812 19:11:44.602927 20776 consensus_queue.cc:260] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.605526 20776 ts_tablet_manager.cc:1428] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6: Time spent starting tablet: real 0.048s user 0.006s sys 0.001s
I20260812 19:11:44.608479 20781 ts_tablet_manager.cc:1428] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb: Time spent starting tablet: real 0.035s user 0.012s sys 0.004s
I20260812 19:11:44.610224 20786 raft_consensus.cc:504] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.612358 20777 tablet_bootstrap.cc:654] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.613425 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3016afd35dd4d4288c4d55f403c268e" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
I20260812 19:11:44.614156 20551 raft_consensus.cc:3037] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.617156 20781 tablet_bootstrap.cc:492] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb: Bootstrap starting.
I20260812 19:11:44.621881 20786 leader_election.cc:288] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 1 election: Requested vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.622535 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3016afd35dd4d4288c4d55f403c268e" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:11:44.623246 20723 raft_consensus.cc:3037] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.627453 20781 tablet_bootstrap.cc:654] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb: Neither blocks nor log segments found. Creating new log.
W20260812 19:11:44.631858 20512 consensus_peers.cc:487] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 -> Peer 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591): Couldn't send request to peer 3e47e633f9784af7b18e39250f7211df. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:11:44.634996 20723 raft_consensus.cc:2454] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9645294d712440283be14a35f4e4beb in term 1.
I20260812 19:11:44.635942 20551 raft_consensus.cc:2454] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9645294d712440283be14a35f4e4beb in term 1.
I20260812 19:11:44.635322 20423 catalog_manager.cc:5495] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:44.636762 20595 leader_election.cc:302] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e47e633f9784af7b18e39250f7211df, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:11:44.637686 20786 raft_consensus.cc:2781] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:11:44.638953 20786 raft_consensus.cc:686] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 1 LEADER]: Becoming Leader. State: Replica: c9645294d712440283be14a35f4e4beb, State: Running, Role: LEADER
I20260812 19:11:44.639859 20786 consensus_queue.cc:237] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.642295 20777 tablet_bootstrap.cc:492] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df: No bootstrap required, opened a new log
I20260812 19:11:44.642827 20777 ts_tablet_manager.cc:1397] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.003s
I20260812 19:11:44.645175 20777 raft_consensus.cc:348] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.646054 20777 raft_consensus.cc:374] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.646319 20777 raft_consensus.cc:729] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.647167 20777 consensus_queue.cc:260] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.649014 20781 tablet_bootstrap.cc:492] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb: No bootstrap required, opened a new log
I20260812 19:11:44.649456 20781 ts_tablet_manager.cc:1397] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.008s
I20260812 19:11:44.651455 20419 catalog_manager.cc:5495] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb reported cstate change: term changed from 0 to 1, leader changed from <none> to c9645294d712440283be14a35f4e4beb (127.19.70.130). New cstate: current_term: 1 leader_uuid: "c9645294d712440283be14a35f4e4beb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:44.652392 20777 ts_tablet_manager.cc:1428] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260812 19:11:44.652326 20781 raft_consensus.cc:348] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.653146 20781 raft_consensus.cc:374] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.653450 20781 raft_consensus.cc:729] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9645294d712440283be14a35f4e4beb, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.654258 20781 consensus_queue.cc:260] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.656915 20781 ts_tablet_manager.cc:1428] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 19:11:44.657866 20781 tablet_bootstrap.cc:492] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb: Bootstrap starting.
I20260812 19:11:44.662599 20781 tablet_bootstrap.cc:654] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb: Neither blocks nor log segments found. Creating new log.
I20260812 19:11:44.669627 20781 tablet_bootstrap.cc:492] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb: No bootstrap required, opened a new log
I20260812 19:11:44.670073 20781 ts_tablet_manager.cc:1397] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260812 19:11:44.672044 20781 raft_consensus.cc:348] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.672644 20781 raft_consensus.cc:374] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:11:44.672873 20781 raft_consensus.cc:729] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9645294d712440283be14a35f4e4beb, State: Initialized, Role: FOLLOWER
I20260812 19:11:44.673390 20781 consensus_queue.cc:260] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.675256 20781 ts_tablet_manager.cc:1428] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 19:11:44.678676 20786 raft_consensus.cc:482] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:44.679301 20780 consensus_queue.cc:1035] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:11:44.679214 20786 raft_consensus.cc:504] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.681368 20786 leader_election.cc:288] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.682041 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0066afe62a194830a36fd1a214ea579a" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" is_pre_election: true
I20260812 19:11:44.682492 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0066afe62a194830a36fd1a214ea579a" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df" is_pre_election: true
I20260812 19:11:44.684119 20595 leader_election.cc:302] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9645294d712440283be14a35f4e4beb; no voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df
I20260812 19:11:44.685704 20849 raft_consensus.cc:2726] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:11:44.686347 20849 raft_consensus.cc:482] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:44.686812 20849 raft_consensus.cc:504] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.688998 20849 leader_election.cc:288] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.690601 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" is_pre_election: true
I20260812 19:11:44.690900 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df" is_pre_election: true
I20260812 19:11:44.691124 20849 raft_consensus.cc:482] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:44.691628 20723 raft_consensus.cc:2379] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9645294d712440283be14a35f4e4beb in current term 1: Already voted for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in this term.
I20260812 19:11:44.692947 20595 leader_election.cc:302] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9645294d712440283be14a35f4e4beb; no voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df
I20260812 19:11:44.693821 20786 raft_consensus.cc:3037] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.691644 20849 raft_consensus.cc:504] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.695815 20849 leader_election.cc:288] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.696646 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" is_pre_election: true
I20260812 19:11:44.698766 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e47e633f9784af7b18e39250f7211df" is_pre_election: true
I20260812 19:11:44.699445 20723 raft_consensus.cc:2379] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9645294d712440283be14a35f4e4beb in current term 1: Already voted for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in this term.
I20260812 19:11:44.700560 20595 leader_election.cc:302] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9645294d712440283be14a35f4e4beb; no voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df
I20260812 19:11:44.701326 20849 raft_consensus.cc:3037] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.703684 20786 raft_consensus.cc:2726] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:11:44.707907 20849 raft_consensus.cc:2726] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:11:44.711043 20780 consensus_queue.cc:1035] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:11:44.725592 20636 raft_consensus.cc:3037] T 147c4ad62d7b4714815454a49c2abd7b P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.727476 20723 raft_consensus.cc:3037] T 66f8b8ae72814bfbbf087d4847f8cde7 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.734921 20797 consensus_queue.cc:1035] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:11:44.736110 20795 consensus_queue.cc:1035] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:11:44.738055 20795 consensus_queue.cc:1035] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:11:44.761565 20720 raft_consensus.cc:3037] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.766317 20797 consensus_queue.cc:1035] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:11:44.773298 20797 consensus_queue.cc:1035] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:11:44.775501 20636 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e3016afd35dd4d4288c4d55f403c268e"
dest_uuid: "c9645294d712440283be14a35f4e4beb"
mode: GRACEFUL
new_leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
from {username='slave'} at 127.0.0.1:39286
I20260812 19:11:44.776343 20636 raft_consensus.cc:595] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 1 LEADER]: Received request to transfer leadership to TS 2862c0a6eacd49d4a3982d8169e35fa6
I20260812 19:11:44.805887 20723 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599"
dest_uuid: "3e47e633f9784af7b18e39250f7211df"
mode: GRACEFUL
new_leader_uuid: "c9645294d712440283be14a35f4e4beb"
from {username='slave'} at 127.0.0.1:55592
I20260812 19:11:44.806551 20723 raft_consensus.cc:595] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 1 LEADER]: Received request to transfer leadership to TS c9645294d712440283be14a35f4e4beb
I20260812 19:11:44.821686 20551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365"
dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
mode: GRACEFUL
new_leader_uuid: "c9645294d712440283be14a35f4e4beb"
from {username='slave'} at 127.0.0.1:40220
I20260812 19:11:44.822300 20551 raft_consensus.cc:595] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Received request to transfer leadership to TS c9645294d712440283be14a35f4e4beb
I20260812 19:11:44.827653 20551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "02bf63ed4d474c0a9b03bde1a2aa2f7a"
dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
mode: GRACEFUL
new_leader_uuid: "c9645294d712440283be14a35f4e4beb"
from {username='slave'} at 127.0.0.1:40220
I20260812 19:11:44.828222 20551 raft_consensus.cc:595] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Received request to transfer leadership to TS c9645294d712440283be14a35f4e4beb
I20260812 19:11:44.833154 20551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0066afe62a194830a36fd1a214ea579a"
dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
mode: GRACEFUL
new_leader_uuid: "c9645294d712440283be14a35f4e4beb"
from {username='slave'} at 127.0.0.1:40220
I20260812 19:11:44.833751 20551 raft_consensus.cc:595] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Received request to transfer leadership to TS c9645294d712440283be14a35f4e4beb
I20260812 19:11:44.839020 20551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "40269a4340dd4476beeface532a7d1d0"
dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
mode: GRACEFUL
new_leader_uuid: "3e47e633f9784af7b18e39250f7211df"
from {username='slave'} at 127.0.0.1:40220
I20260812 19:11:44.839617 20551 raft_consensus.cc:595] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Received request to transfer leadership to TS 3e47e633f9784af7b18e39250f7211df
I20260812 19:11:44.862177 20797 consensus_queue.cc:1035] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:11:44.873473 20798 consensus_queue.cc:1035] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:11:44.880061 20784 raft_consensus.cc:482] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:11:44.880630 20784 raft_consensus.cc:504] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.882959 20784 leader_election.cc:288] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), c9645294d712440283be14a35f4e4beb (127.19.70.130:39101)
I20260812 19:11:44.883729 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bf63ed4d474c0a9b03bde1a2aa2f7a" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" is_pre_election: true
I20260812 19:11:44.885124 20635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bf63ed4d474c0a9b03bde1a2aa2f7a" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9645294d712440283be14a35f4e4beb" is_pre_election: true
I20260812 19:11:44.885602 20636 raft_consensus.cc:3037] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:44.886602 20680 leader_election.cc:302] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e47e633f9784af7b18e39250f7211df; no voters: 2862c0a6eacd49d4a3982d8169e35fa6, c9645294d712440283be14a35f4e4beb
I20260812 19:11:44.887403 20784 raft_consensus.cc:2726] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:11:44.896732 20797 raft_consensus.cc:979] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6: : Instructing follower c9645294d712440283be14a35f4e4beb to start an election
I20260812 19:11:44.897189 20797 raft_consensus.cc:1067] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Signalling peer c9645294d712440283be14a35f4e4beb to start an election
I20260812 19:11:44.898824 20636 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0066afe62a194830a36fd1a214ea579a"
dest_uuid: "c9645294d712440283be14a35f4e4beb"
from {username='slave'} at 127.0.0.1:39264
I20260812 19:11:44.899338 20636 raft_consensus.cc:482] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:44.899660 20636 raft_consensus.cc:3037] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:44.904628 20636 raft_consensus.cc:504] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.906634 20636 leader_election.cc:288] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 2 election: Requested vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.907555 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0066afe62a194830a36fd1a214ea579a" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
I20260812 19:11:44.907876 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0066afe62a194830a36fd1a214ea579a" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:11:44.908291 20551 raft_consensus.cc:3032] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:11:44.908607 20723 raft_consensus.cc:3037] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:44.908651 20551 raft_consensus.cc:729] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:44.909569 20551 consensus_queue.cc:260] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.911151 20551 raft_consensus.cc:3037] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:44.914940 20723 raft_consensus.cc:2454] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9645294d712440283be14a35f4e4beb in term 2.
I20260812 19:11:44.916069 20595 leader_election.cc:302] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e47e633f9784af7b18e39250f7211df, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:11:44.916728 20551 raft_consensus.cc:2454] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9645294d712440283be14a35f4e4beb in term 2.
I20260812 19:11:44.916857 20849 raft_consensus.cc:2781] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:11:44.917649 20849 raft_consensus.cc:686] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [term 2 LEADER]: Becoming Leader. State: Replica: c9645294d712440283be14a35f4e4beb, State: Running, Role: LEADER
I20260812 19:11:44.918534 20849 consensus_queue.cc:237] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.925247 20423 catalog_manager.cc:5495] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb reported cstate change: term changed from 1 to 2, leader changed from 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129) to c9645294d712440283be14a35f4e4beb (127.19.70.130). New cstate: current_term: 2 leader_uuid: "c9645294d712440283be14a35f4e4beb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:44.945291 20784 consensus_queue.cc:1035] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [LEADER]: Connected to new peer: Peer: permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:11:44.957074 20784 consensus_queue.cc:1035] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:11:44.966130 20830 raft_consensus.cc:979] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df: : Instructing follower c9645294d712440283be14a35f4e4beb to start an election
I20260812 19:11:44.966785 20870 raft_consensus.cc:1067] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 1 LEADER]: Signalling peer c9645294d712440283be14a35f4e4beb to start an election
I20260812 19:11:44.969554 20635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599"
dest_uuid: "c9645294d712440283be14a35f4e4beb"
from {username='slave'} at 127.0.0.1:39274
I20260812 19:11:44.969645 20798 consensus_queue.cc:1035] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:11:44.970218 20635 raft_consensus.cc:482] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:44.970686 20635 raft_consensus.cc:3037] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:44.976518 20635 raft_consensus.cc:504] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.978645 20635 leader_election.cc:288] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 2 election: Requested vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:44.980279 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
I20260812 19:11:44.980947 20551 raft_consensus.cc:3037] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:44.983403 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:11:44.984074 20723 raft_consensus.cc:3032] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:11:44.984380 20723 raft_consensus.cc:729] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Running, Role: LEADER
I20260812 19:11:44.985176 20723 consensus_queue.cc:260] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:44.986513 20723 raft_consensus.cc:3037] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:44.988587 20551 raft_consensus.cc:2454] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9645294d712440283be14a35f4e4beb in term 2.
I20260812 19:11:44.989621 20798 consensus_queue.cc:1035] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:11:44.992103 20595 leader_election.cc:302] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:11:44.993261 20786 raft_consensus.cc:2781] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:11:44.993841 20786 raft_consensus.cc:686] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 2 LEADER]: Becoming Leader. State: Replica: c9645294d712440283be14a35f4e4beb, State: Running, Role: LEADER
I20260812 19:11:44.995007 20786 consensus_queue.cc:237] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:45.001435 20797 raft_consensus.cc:979] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6: : Instructing follower c9645294d712440283be14a35f4e4beb to start an election
I20260812 19:11:45.001435 20723 raft_consensus.cc:2454] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9645294d712440283be14a35f4e4beb in term 2.
I20260812 19:11:45.002074 20797 raft_consensus.cc:1067] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Signalling peer c9645294d712440283be14a35f4e4beb to start an election
I20260812 19:11:45.003966 20636 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365"
dest_uuid: "c9645294d712440283be14a35f4e4beb"
from {username='slave'} at 127.0.0.1:39264
I20260812 19:11:45.004701 20636 raft_consensus.cc:482] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:45.005110 20636 raft_consensus.cc:3037] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:45.007105 20419 catalog_manager.cc:5495] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb reported cstate change: term changed from 1 to 2, leader changed from 3e47e633f9784af7b18e39250f7211df (127.19.70.131) to c9645294d712440283be14a35f4e4beb (127.19.70.130). New cstate: current_term: 2 leader_uuid: "c9645294d712440283be14a35f4e4beb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:45.012581 20636 raft_consensus.cc:504] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:45.014792 20636 leader_election.cc:288] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 2 election: Requested vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:45.016059 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
I20260812 19:11:45.016780 20551 raft_consensus.cc:3032] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:11:45.017122 20551 raft_consensus.cc:729] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:45.017232 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:11:45.017930 20723 raft_consensus.cc:3037] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:45.017975 20551 consensus_queue.cc:260] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:45.022249 20551 raft_consensus.cc:3037] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:45.026151 20723 raft_consensus.cc:2454] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9645294d712440283be14a35f4e4beb in term 2.
I20260812 19:11:45.027618 20595 leader_election.cc:302] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e47e633f9784af7b18e39250f7211df, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:11:45.028520 20786 raft_consensus.cc:2781] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:11:45.029062 20786 raft_consensus.cc:686] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 2 LEADER]: Becoming Leader. State: Replica: c9645294d712440283be14a35f4e4beb, State: Running, Role: LEADER
I20260812 19:11:45.029949 20786 consensus_queue.cc:237] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:45.030678 20551 raft_consensus.cc:2454] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9645294d712440283be14a35f4e4beb in term 2.
I20260812 19:11:45.038591 20419 catalog_manager.cc:5495] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb reported cstate change: term changed from 1 to 2, leader changed from 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129) to c9645294d712440283be14a35f4e4beb (127.19.70.130). New cstate: current_term: 2 leader_uuid: "c9645294d712440283be14a35f4e4beb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:45.109856 20786 consensus_queue.cc:1035] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:11:45.117233 20849 raft_consensus.cc:979] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb: : Instructing follower 2862c0a6eacd49d4a3982d8169e35fa6 to start an election
I20260812 19:11:45.117906 20786 raft_consensus.cc:1067] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 1 LEADER]: Signalling peer 2862c0a6eacd49d4a3982d8169e35fa6 to start an election
I20260812 19:11:45.119976 20550 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e3016afd35dd4d4288c4d55f403c268e"
dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
from {username='slave'} at 127.0.0.1:40206
I20260812 19:11:45.121037 20550 raft_consensus.cc:482] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:45.121464 20550 raft_consensus.cc:3037] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:45.124627 20786 consensus_queue.cc:1035] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:11:45.131076 20550 raft_consensus.cc:504] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:45.139065 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3016afd35dd4d4288c4d55f403c268e" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:45.139892 20636 raft_consensus.cc:3032] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:11:45.140314 20636 raft_consensus.cc:729] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9645294d712440283be14a35f4e4beb, State: Running, Role: LEADER
I20260812 19:11:45.141793 20550 leader_election.cc:288] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 2 election: Requested vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:45.141263 20636 consensus_queue.cc:260] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:45.143097 20636 raft_consensus.cc:3037] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:45.143733 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3016afd35dd4d4288c4d55f403c268e" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:11:45.144495 20723 raft_consensus.cc:3037] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:45.152480 20723 raft_consensus.cc:2454] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 2.
I20260812 19:11:45.153180 20636 raft_consensus.cc:2454] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 2.
I20260812 19:11:45.154374 20512 leader_election.cc:302] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df; no voters:
I20260812 19:11:45.155809 20797 raft_consensus.cc:2781] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:11:45.156491 20797 raft_consensus.cc:686] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 LEADER]: Becoming Leader. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:45.157800 20797 consensus_queue.cc:237] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:45.168576 20419 catalog_manager.cc:5495] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 reported cstate change: term changed from 1 to 2, leader changed from c9645294d712440283be14a35f4e4beb (127.19.70.130) to 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129). New cstate: current_term: 2 leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:45.187330 20797 consensus_queue.cc:1035] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:11:45.203677 20798 consensus_queue.cc:1035] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:11:45.225325 20795 raft_consensus.cc:979] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6: : Instructing follower 3e47e633f9784af7b18e39250f7211df to start an election
I20260812 19:11:45.227175 20780 raft_consensus.cc:1067] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Signalling peer 3e47e633f9784af7b18e39250f7211df to start an election
I20260812 19:11:45.243363 20720 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40269a4340dd4476beeface532a7d1d0"
dest_uuid: "3e47e633f9784af7b18e39250f7211df"
from {username='slave'} at 127.0.0.1:55570
I20260812 19:11:45.250950 20723 raft_consensus.cc:3037] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:11:45.252570 20720 raft_consensus.cc:482] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:45.253160 20720 raft_consensus.cc:3037] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:45.262254 20720 raft_consensus.cc:504] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:45.265024 20720 leader_election.cc:288] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 2 election: Requested vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), c9645294d712440283be14a35f4e4beb (127.19.70.130:39101)
I20260812 19:11:45.266695 20798 raft_consensus.cc:979] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6: : Instructing follower c9645294d712440283be14a35f4e4beb to start an election
I20260812 19:11:45.266834 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40269a4340dd4476beeface532a7d1d0" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:45.267701 20798 raft_consensus.cc:1067] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Signalling peer c9645294d712440283be14a35f4e4beb to start an election
I20260812 19:11:45.267961 20636 raft_consensus.cc:3037] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:45.268988 20550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40269a4340dd4476beeface532a7d1d0" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
I20260812 19:11:45.269838 20550 raft_consensus.cc:3032] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:11:45.270407 20550 raft_consensus.cc:729] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:45.272113 20550 consensus_queue.cc:260] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:45.277155 20635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "02bf63ed4d474c0a9b03bde1a2aa2f7a"
dest_uuid: "c9645294d712440283be14a35f4e4beb"
from {username='slave'} at 127.0.0.1:39264
I20260812 19:11:45.278091 20635 raft_consensus.cc:482] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:45.274245 20550 raft_consensus.cc:3037] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:45.278656 20635 raft_consensus.cc:3037] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:45.280587 20636 raft_consensus.cc:2454] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e47e633f9784af7b18e39250f7211df in term 2.
I20260812 19:11:45.282160 20680 leader_election.cc:302] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e47e633f9784af7b18e39250f7211df, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:11:45.283491 20784 raft_consensus.cc:2781] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:11:45.284078 20784 raft_consensus.cc:686] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df [term 2 LEADER]: Becoming Leader. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Running, Role: LEADER
I20260812 19:11:45.286593 20784 consensus_queue.cc:237] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:45.308501 20635 raft_consensus.cc:504] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:45.312139 20635 leader_election.cc:288] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 2 election: Requested vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:45.313506 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bf63ed4d474c0a9b03bde1a2aa2f7a" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:11:45.314231 20723 raft_consensus.cc:3037] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:45.315892 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bf63ed4d474c0a9b03bde1a2aa2f7a" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
I20260812 19:11:45.316902 20551 raft_consensus.cc:3032] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:11:45.317376 20551 raft_consensus.cc:729] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:45.318920 20551 consensus_queue.cc:260] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:45.319531 20550 raft_consensus.cc:2454] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e47e633f9784af7b18e39250f7211df in term 2.
I20260812 19:11:45.320716 20551 raft_consensus.cc:3037] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:45.325531 20723 raft_consensus.cc:2454] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9645294d712440283be14a35f4e4beb in term 2.
I20260812 19:11:45.327406 20595 leader_election.cc:302] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e47e633f9784af7b18e39250f7211df, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:11:45.328545 20786 raft_consensus.cc:2781] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:11:45.329416 20786 raft_consensus.cc:686] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 2 LEADER]: Becoming Leader. State: Replica: c9645294d712440283be14a35f4e4beb, State: Running, Role: LEADER
I20260812 19:11:45.330914 20786 consensus_queue.cc:237] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:45.333257 20551 raft_consensus.cc:2454] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9645294d712440283be14a35f4e4beb in term 2.
I20260812 19:11:45.337818 20421 catalog_manager.cc:5495] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df reported cstate change: term changed from 1 to 2, leader changed from 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129) to 3e47e633f9784af7b18e39250f7211df (127.19.70.131). New cstate: current_term: 2 leader_uuid: "3e47e633f9784af7b18e39250f7211df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: HEALTHY } } }
I20260812 19:11:45.356993 20419 catalog_manager.cc:5495] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb reported cstate change: term changed from 1 to 2, leader changed from 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129) to c9645294d712440283be14a35f4e4beb (127.19.70.130). New cstate: current_term: 2 leader_uuid: "c9645294d712440283be14a35f4e4beb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:45.394759 20723 raft_consensus.cc:1261] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Refusing update from remote peer c9645294d712440283be14a35f4e4beb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:45.396082 20786 consensus_queue.cc:1035] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:11:45.405097 20551 raft_consensus.cc:1261] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Refusing update from remote peer c9645294d712440283be14a35f4e4beb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:45.407181 20786 consensus_queue.cc:1035] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:45.435187 20551 raft_consensus.cc:1261] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Refusing update from remote peer c9645294d712440283be14a35f4e4beb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:45.437141 20786 consensus_queue.cc:1035] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:45.449609 20723 raft_consensus.cc:1261] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Refusing update from remote peer c9645294d712440283be14a35f4e4beb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:45.457324 20882 consensus_queue.cc:1035] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:45.479997 20723 raft_consensus.cc:1261] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Refusing update from remote peer c9645294d712440283be14a35f4e4beb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:45.481930 20882 consensus_queue.cc:1035] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:11:45.493049 20551 raft_consensus.cc:1261] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Refusing update from remote peer c9645294d712440283be14a35f4e4beb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:45.496615 20882 consensus_queue.cc:1035] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:45.646160 20635 raft_consensus.cc:1261] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Refusing update from remote peer 2862c0a6eacd49d4a3982d8169e35fa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:45.647864 20880 consensus_queue.cc:1035] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:11:45.662892 20723 raft_consensus.cc:1261] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Refusing update from remote peer 2862c0a6eacd49d4a3982d8169e35fa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:45.665249 20795 consensus_queue.cc:1035] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:45.691313 20551 raft_consensus.cc:1261] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Refusing update from remote peer 3e47e633f9784af7b18e39250f7211df: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:45.692836 20784 consensus_queue.cc:1035] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df [LEADER]: Connected to new peer: Peer: permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:45.704713 20635 raft_consensus.cc:1261] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Refusing update from remote peer 3e47e633f9784af7b18e39250f7211df: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:45.712194 20784 consensus_queue.cc:1035] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:45.775732 20551 raft_consensus.cc:1261] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Refusing update from remote peer c9645294d712440283be14a35f4e4beb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:45.777062 20882 consensus_queue.cc:1035] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:11:45.786039 20723 raft_consensus.cc:1261] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Refusing update from remote peer c9645294d712440283be14a35f4e4beb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:45.787678 20786 consensus_queue.cc:1035] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:11:47.875746 19738 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260812 19:11:47.876461 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:47.904285 20551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed"
dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
mode: GRACEFUL
new_leader_uuid: "3e47e633f9784af7b18e39250f7211df"
from {username='slave'} at 127.0.0.1:38884
I20260812 19:11:47.904888 20551 raft_consensus.cc:595] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Received request to transfer leadership to TS 3e47e633f9784af7b18e39250f7211df
I20260812 19:11:47.907580 20551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e3016afd35dd4d4288c4d55f403c268e"
dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
mode: GRACEFUL
new_leader_uuid: "3e47e633f9784af7b18e39250f7211df"
from {username='slave'} at 127.0.0.1:38884
I20260812 19:11:47.908144 20551 raft_consensus.cc:595] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 LEADER]: Received request to transfer leadership to TS 3e47e633f9784af7b18e39250f7211df
I20260812 19:11:47.909083 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260812 19:11:47.909574 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:48.154157 20886 raft_consensus.cc:979] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6: : Instructing follower 3e47e633f9784af7b18e39250f7211df to start an election
I20260812 19:11:48.154760 20886 raft_consensus.cc:1067] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Signalling peer 3e47e633f9784af7b18e39250f7211df to start an election
I20260812 19:11:48.156842 20720 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed"
dest_uuid: "3e47e633f9784af7b18e39250f7211df"
from {username='slave'} at 127.0.0.1:55570
I20260812 19:11:48.157395 20720 raft_consensus.cc:482] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:48.157747 20720 raft_consensus.cc:3037] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:48.163836 20720 raft_consensus.cc:504] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:48.165624 20720 leader_election.cc:288] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 2 election: Requested vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), c9645294d712440283be14a35f4e4beb (127.19.70.130:39101)
I20260812 19:11:48.166767 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:48.166759 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
I20260812 19:11:48.167470 20636 raft_consensus.cc:3037] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:48.167532 20551 raft_consensus.cc:3032] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:11:48.167932 20551 raft_consensus.cc:729] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:48.168692 20551 consensus_queue.cc:260] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:48.169858 20551 raft_consensus.cc:3037] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:11:48.173631 20636 raft_consensus.cc:2454] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e47e633f9784af7b18e39250f7211df in term 2.
I20260812 19:11:48.174999 20680 leader_election.cc:302] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e47e633f9784af7b18e39250f7211df, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:11:48.175781 20551 raft_consensus.cc:2454] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e47e633f9784af7b18e39250f7211df in term 2.
I20260812 19:11:48.175858 20784 raft_consensus.cc:2781] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:11:48.177256 20784 raft_consensus.cc:686] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 2 LEADER]: Becoming Leader. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Running, Role: LEADER
I20260812 19:11:48.178099 20784 consensus_queue.cc:237] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:48.186381 20421 catalog_manager.cc:5495] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df reported cstate change: term changed from 1 to 2, leader changed from 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129) to 3e47e633f9784af7b18e39250f7211df (127.19.70.131). New cstate: current_term: 2 leader_uuid: "3e47e633f9784af7b18e39250f7211df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: HEALTHY } } }
I20260812 19:11:48.244848 20886 raft_consensus.cc:979] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6: : Instructing follower 3e47e633f9784af7b18e39250f7211df to start an election
I20260812 19:11:48.245323 20890 raft_consensus.cc:1067] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 LEADER]: Signalling peer 3e47e633f9784af7b18e39250f7211df to start an election
I20260812 19:11:48.247138 20720 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e3016afd35dd4d4288c4d55f403c268e"
dest_uuid: "3e47e633f9784af7b18e39250f7211df"
from {username='slave'} at 127.0.0.1:55570
I20260812 19:11:48.247637 20720 raft_consensus.cc:482] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:48.247905 20720 raft_consensus.cc:3037] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:48.253412 20720 raft_consensus.cc:504] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:48.255869 20720 leader_election.cc:288] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 3 election: Requested vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), c9645294d712440283be14a35f4e4beb (127.19.70.130:39101)
I20260812 19:11:48.256595 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3016afd35dd4d4288c4d55f403c268e" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:48.257223 20636 raft_consensus.cc:3037] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:48.257324 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3016afd35dd4d4288c4d55f403c268e" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
I20260812 19:11:48.258049 20551 raft_consensus.cc:3032] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:11:48.258409 20551 raft_consensus.cc:729] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:48.259338 20551 consensus_queue.cc:260] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:48.260460 20551 raft_consensus.cc:3037] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:48.263370 20636 raft_consensus.cc:2454] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e47e633f9784af7b18e39250f7211df in term 3.
I20260812 19:11:48.264609 20680 leader_election.cc:302] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e47e633f9784af7b18e39250f7211df, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:11:48.265522 20784 raft_consensus.cc:2781] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:11:48.265966 20784 raft_consensus.cc:686] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 3 LEADER]: Becoming Leader. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Running, Role: LEADER
I20260812 19:11:48.266476 20551 raft_consensus.cc:2454] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e47e633f9784af7b18e39250f7211df in term 3.
I20260812 19:11:48.266820 20784 consensus_queue.cc:237] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:48.274009 20421 catalog_manager.cc:5495] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df reported cstate change: term changed from 2 to 3, leader changed from 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129) to 3e47e633f9784af7b18e39250f7211df (127.19.70.131). New cstate: current_term: 3 leader_uuid: "3e47e633f9784af7b18e39250f7211df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: HEALTHY } } }
I20260812 19:11:48.607801 20636 raft_consensus.cc:1261] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Refusing update from remote peer 3e47e633f9784af7b18e39250f7211df: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:48.609576 20784 consensus_queue.cc:1035] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:48.625975 20551 raft_consensus.cc:1261] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Refusing update from remote peer 3e47e633f9784af7b18e39250f7211df: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:11:48.628608 20893 consensus_queue.cc:1035] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [LEADER]: Connected to new peer: Peer: permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:11:48.667975 20636 raft_consensus.cc:1261] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Refusing update from remote peer 3e47e633f9784af7b18e39250f7211df: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:48.669646 20905 consensus_queue.cc:1035] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:11:48.670114 20551 raft_consensus.cc:1261] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 FOLLOWER]: Refusing update from remote peer 3e47e633f9784af7b18e39250f7211df: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:48.671643 20905 consensus_queue.cc:1035] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [LEADER]: Connected to new peer: Peer: permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:11:48.911158 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:48.918753 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:48.919260 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:49.920152 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:49.926080 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:49.926604 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:50.927502 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:50.933175 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:50.933743 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:51.934737 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:51.939992 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:51.940536 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:52.941452 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:52.947113 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:52.947603 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:53.948493 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:53.953915 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:53.954455 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:54.955488 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:54.962049 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:54.962750 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:56.112589 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:56.122237 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:56.123035 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:11:57.124094 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:11:57.129313 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:11:57.129846 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260812 19:11:57.189970 20753 debug-util.cc:398] Leaking SignalData structure 0x7b08000ec6c0 after lost signal to thread 19742
W20260812 19:11:57.191028 20753 debug-util.cc:398] Leaking SignalData structure 0x7b08000ecda0 after lost signal to thread 20467
W20260812 19:11:57.192049 20753 debug-util.cc:398] Leaking SignalData structure 0x7b08000ab420 after lost signal to thread 20578
W20260812 19:11:57.193135 20753 debug-util.cc:398] Leaking SignalData structure 0x7b0800100160 after lost signal to thread 20663
W20260812 19:11:57.194118 20753 debug-util.cc:398] Leaking SignalData structure 0x7b08001f65e0 after lost signal to thread 20756
I20260812 19:11:58.181527 19738 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260812 19:11:58.193672 20720 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e3016afd35dd4d4288c4d55f403c268e"
dest_uuid: "3e47e633f9784af7b18e39250f7211df"
mode: GRACEFUL
new_leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
from {username='slave'} at 127.0.0.1:55592
I20260812 19:11:58.194732 20720 raft_consensus.cc:595] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 3 LEADER]: Received request to transfer leadership to TS 2862c0a6eacd49d4a3982d8169e35fa6
I20260812 19:11:58.202443 20720 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed"
dest_uuid: "3e47e633f9784af7b18e39250f7211df"
mode: GRACEFUL
new_leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
from {username='slave'} at 127.0.0.1:55592
I20260812 19:11:58.203055 20720 raft_consensus.cc:595] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 2 LEADER]: Received request to transfer leadership to TS 2862c0a6eacd49d4a3982d8169e35fa6
I20260812 19:11:58.218129 20636 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599"
dest_uuid: "c9645294d712440283be14a35f4e4beb"
mode: GRACEFUL
new_leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
from {username='slave'} at 127.0.0.1:39286
I20260812 19:11:58.218899 20636 raft_consensus.cc:595] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 2 LEADER]: Received request to transfer leadership to TS 2862c0a6eacd49d4a3982d8169e35fa6
I20260812 19:11:58.243420 20636 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365"
dest_uuid: "c9645294d712440283be14a35f4e4beb"
mode: GRACEFUL
new_leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
from {username='slave'} at 127.0.0.1:39286
I20260812 19:11:58.244240 20636 raft_consensus.cc:595] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 2 LEADER]: Received request to transfer leadership to TS 2862c0a6eacd49d4a3982d8169e35fa6
I20260812 19:11:58.253149 20928 raft_consensus.cc:979] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb: : Instructing follower 2862c0a6eacd49d4a3982d8169e35fa6 to start an election
I20260812 19:11:58.253875 20927 raft_consensus.cc:1067] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 2 LEADER]: Signalling peer 2862c0a6eacd49d4a3982d8169e35fa6 to start an election
I20260812 19:11:58.256129 20636 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "02bf63ed4d474c0a9b03bde1a2aa2f7a"
dest_uuid: "c9645294d712440283be14a35f4e4beb"
mode: GRACEFUL
new_leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
from {username='slave'} at 127.0.0.1:39286
I20260812 19:11:58.256690 20636 raft_consensus.cc:595] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 2 LEADER]: Received request to transfer leadership to TS 2862c0a6eacd49d4a3982d8169e35fa6
I20260812 19:11:58.259434 20550 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365"
dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
from {username='slave'} at 127.0.0.1:40206
I20260812 19:11:58.260391 20550 raft_consensus.cc:482] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:58.261058 20550 raft_consensus.cc:3037] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:58.267987 20929 raft_consensus.cc:979] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb: : Instructing follower 2862c0a6eacd49d4a3982d8169e35fa6 to start an election
I20260812 19:11:58.268814 20927 raft_consensus.cc:1067] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 2 LEADER]: Signalling peer 2862c0a6eacd49d4a3982d8169e35fa6 to start an election
I20260812 19:11:58.271049 20551 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599"
dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
from {username='slave'} at 127.0.0.1:40206
I20260812 19:11:58.271943 20551 raft_consensus.cc:482] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:58.272445 20551 raft_consensus.cc:3037] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:58.279817 20550 raft_consensus.cc:504] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:58.283958 20550 leader_election.cc:288] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 3 election: Requested vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:58.286092 20720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:11:58.286443 20551 raft_consensus.cc:504] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:58.287002 20720 raft_consensus.cc:3037] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:58.288820 20551 leader_election.cc:288] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 3 election: Requested vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:58.290722 20635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:58.291589 20635 raft_consensus.cc:3032] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:11:58.291987 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:11:58.292090 20635 raft_consensus.cc:729] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c9645294d712440283be14a35f4e4beb, State: Running, Role: LEADER
I20260812 19:11:58.292668 20723 raft_consensus.cc:3037] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:58.293488 20635 consensus_queue.cc:260] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:58.286093 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:58.295958 20636 raft_consensus.cc:3032] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:11:58.296272 20636 raft_consensus.cc:729] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c9645294d712440283be14a35f4e4beb, State: Running, Role: LEADER
I20260812 19:11:58.297151 20636 consensus_queue.cc:260] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:58.298111 20720 raft_consensus.cc:2454] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 3.
I20260812 19:11:58.300292 20512 leader_election.cc:302] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df; no voters:
I20260812 19:11:58.302107 20890 raft_consensus.cc:2781] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:11:58.302754 20890 raft_consensus.cc:686] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 LEADER]: Becoming Leader. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:58.303978 20890 consensus_queue.cc:237] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:58.307967 20723 raft_consensus.cc:2454] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 3.
I20260812 19:11:58.308075 20636 raft_consensus.cc:3037] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:58.309561 20635 raft_consensus.cc:3037] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:58.314646 20512 leader_election.cc:302] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df; no voters:
I20260812 19:11:58.318557 20930 raft_consensus.cc:2781] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:11:58.319619 20930 raft_consensus.cc:686] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 LEADER]: Becoming Leader. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:58.321904 20930 consensus_queue.cc:237] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:58.329818 20636 raft_consensus.cc:2454] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 3.
I20260812 19:11:58.334009 20635 raft_consensus.cc:2454] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 3.
I20260812 19:11:58.341455 20419 catalog_manager.cc:5495] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 reported cstate change: term changed from 2 to 3, leader changed from c9645294d712440283be14a35f4e4beb (127.19.70.130) to 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:58.346386 20926 raft_consensus.cc:979] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df: : Instructing follower 2862c0a6eacd49d4a3982d8169e35fa6 to start an election
I20260812 19:11:58.347523 20926 raft_consensus.cc:1067] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 2 LEADER]: Signalling peer 2862c0a6eacd49d4a3982d8169e35fa6 to start an election
I20260812 19:11:58.350010 20551 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed"
dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
from {username='slave'} at 127.0.0.1:40190
I20260812 19:11:58.350920 20551 raft_consensus.cc:482] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:58.351385 20551 raft_consensus.cc:3037] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:58.376233 20551 raft_consensus.cc:504] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:58.380896 20635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:58.381760 20635 raft_consensus.cc:3037] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:58.392796 20635 raft_consensus.cc:2454] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 3.
I20260812 19:11:58.394207 20551 leader_election.cc:288] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 3 election: Requested vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:58.395634 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:11:58.399233 20723 raft_consensus.cc:3032] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:11:58.399631 20723 raft_consensus.cc:729] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Running, Role: LEADER
I20260812 19:11:58.400621 20723 consensus_queue.cc:260] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:58.402138 20723 raft_consensus.cc:3037] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:58.403896 20510 leader_election.cc:302] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:11:58.404814 20930 raft_consensus.cc:2781] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:11:58.405658 20930 raft_consensus.cc:686] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 LEADER]: Becoming Leader. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:58.407801 20930 consensus_queue.cc:237] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:58.413326 20723 raft_consensus.cc:2454] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 3.
I20260812 19:11:58.443136 20419 catalog_manager.cc:5495] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 reported cstate change: term changed from 2 to 3, leader changed from c9645294d712440283be14a35f4e4beb (127.19.70.130) to 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:58.444540 20419 catalog_manager.cc:5495] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 reported cstate change: term changed from 2 to 3, leader changed from 3e47e633f9784af7b18e39250f7211df (127.19.70.131) to 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:58.485556 20926 raft_consensus.cc:979] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df: : Instructing follower 2862c0a6eacd49d4a3982d8169e35fa6 to start an election
I20260812 19:11:58.486778 20934 raft_consensus.cc:1067] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 3 LEADER]: Signalling peer 2862c0a6eacd49d4a3982d8169e35fa6 to start an election
I20260812 19:11:58.489087 20550 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e3016afd35dd4d4288c4d55f403c268e"
dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
from {username='slave'} at 127.0.0.1:40190
I20260812 19:11:58.489971 20550 raft_consensus.cc:482] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:58.490711 20550 raft_consensus.cc:3037] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:11:58.496093 20929 raft_consensus.cc:979] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb: : Instructing follower 2862c0a6eacd49d4a3982d8169e35fa6 to start an election
I20260812 19:11:58.496850 20927 raft_consensus.cc:1067] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 2 LEADER]: Signalling peer 2862c0a6eacd49d4a3982d8169e35fa6 to start an election
I20260812 19:11:58.499006 20551 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "02bf63ed4d474c0a9b03bde1a2aa2f7a"
dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
from {username='slave'} at 127.0.0.1:40206
I20260812 19:11:58.499580 20551 raft_consensus.cc:482] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:11:58.499996 20551 raft_consensus.cc:3037] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:58.500598 20550 raft_consensus.cc:504] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:58.502507 20550 leader_election.cc:288] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 4 election: Requested vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:58.503847 20635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3016afd35dd4d4288c4d55f403c268e" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:58.504426 20635 raft_consensus.cc:3037] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:11:58.504760 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3016afd35dd4d4288c4d55f403c268e" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:11:58.505384 20723 raft_consensus.cc:3032] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:11:58.505694 20723 raft_consensus.cc:729] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Running, Role: LEADER
I20260812 19:11:58.506995 20723 consensus_queue.cc:260] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:58.508469 20723 raft_consensus.cc:3037] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:11:58.508780 20551 raft_consensus.cc:504] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:58.511961 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bf63ed4d474c0a9b03bde1a2aa2f7a" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:11:58.512872 20636 raft_consensus.cc:3032] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:11:58.513259 20636 raft_consensus.cc:729] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c9645294d712440283be14a35f4e4beb, State: Running, Role: LEADER
I20260812 19:11:58.514590 20636 consensus_queue.cc:260] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:58.516812 20636 raft_consensus.cc:3037] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:58.517691 20723 raft_consensus.cc:2454] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 4.
I20260812 19:11:58.519223 20512 leader_election.cc:302] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df; no voters:
I20260812 19:11:58.519793 20551 leader_election.cc:288] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 3 election: Requested vote from peers c9645294d712440283be14a35f4e4beb (127.19.70.130:39101), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:11:58.520964 20930 raft_consensus.cc:2781] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:11:58.521898 20930 raft_consensus.cc:686] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 4 LEADER]: Becoming Leader. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:58.522949 20635 raft_consensus.cc:2454] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 4.
I20260812 19:11:58.523586 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bf63ed4d474c0a9b03bde1a2aa2f7a" candidate_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:11:58.524353 20723 raft_consensus.cc:3037] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:11:58.524968 20930 consensus_queue.cc:237] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:58.533000 20723 raft_consensus.cc:2454] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 3.
I20260812 19:11:58.534343 20512 leader_election.cc:302] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2862c0a6eacd49d4a3982d8169e35fa6, 3e47e633f9784af7b18e39250f7211df; no voters:
I20260812 19:11:58.534771 20636 raft_consensus.cc:2454] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2862c0a6eacd49d4a3982d8169e35fa6 in term 3.
I20260812 19:11:58.536202 20932 raft_consensus.cc:2781] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:11:58.536849 20932 raft_consensus.cc:686] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 LEADER]: Becoming Leader. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:11:58.539409 20932 consensus_queue.cc:237] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:11:58.572077 20419 catalog_manager.cc:5495] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 reported cstate change: term changed from 3 to 4, leader changed from 3e47e633f9784af7b18e39250f7211df (127.19.70.131) to 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129). New cstate: current_term: 4 leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:58.601119 20419 catalog_manager.cc:5495] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 reported cstate change: term changed from 2 to 3, leader changed from c9645294d712440283be14a35f4e4beb (127.19.70.130) to 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:11:58.722823 20723 raft_consensus.cc:1261] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 3 FOLLOWER]: Refusing update from remote peer 2862c0a6eacd49d4a3982d8169e35fa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:58.724427 20930 consensus_queue.cc:1035] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:11:58.755810 20636 raft_consensus.cc:1261] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Refusing update from remote peer 2862c0a6eacd49d4a3982d8169e35fa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:58.757294 20930 consensus_queue.cc:1035] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:11:58.845723 20636 raft_consensus.cc:1261] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Refusing update from remote peer 2862c0a6eacd49d4a3982d8169e35fa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:58.847154 20930 consensus_queue.cc:1035] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:11:58.859247 20723 raft_consensus.cc:1261] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 3 FOLLOWER]: Refusing update from remote peer 2862c0a6eacd49d4a3982d8169e35fa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:58.860819 20930 consensus_queue.cc:1035] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:11:58.914188 20723 raft_consensus.cc:1261] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 3 FOLLOWER]: Refusing update from remote peer 2862c0a6eacd49d4a3982d8169e35fa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:58.915802 20930 consensus_queue.cc:1035] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:11:58.932603 20636 raft_consensus.cc:1261] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Refusing update from remote peer 2862c0a6eacd49d4a3982d8169e35fa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:58.934923 20930 consensus_queue.cc:1035] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:11:58.947683 20636 raft_consensus.cc:1261] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Refusing update from remote peer 2862c0a6eacd49d4a3982d8169e35fa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:58.949558 20932 consensus_queue.cc:1035] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:11:58.961292 20723 raft_consensus.cc:1261] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [term 3 FOLLOWER]: Refusing update from remote peer 2862c0a6eacd49d4a3982d8169e35fa6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:11:58.963521 20932 consensus_queue.cc:1035] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:11:59.095528 20636 raft_consensus.cc:1261] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 4 FOLLOWER]: Refusing update from remote peer 2862c0a6eacd49d4a3982d8169e35fa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:11:59.097307 20890 consensus_queue.cc:1035] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:11:59.109089 20723 raft_consensus.cc:1261] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 4 FOLLOWER]: Refusing update from remote peer 2862c0a6eacd49d4a3982d8169e35fa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:11:59.112138 20890 consensus_queue.cc:1035] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:01.303237 19738 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260812 19:12:01.303926 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:01.311631 20551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "147c4ad62d7b4714815454a49c2abd7b"
dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
mode: GRACEFUL
new_leader_uuid: "3e47e633f9784af7b18e39250f7211df"
from {username='slave'} at 127.0.0.1:38884
I20260812 19:12:01.312410 20551 raft_consensus.cc:595] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Received request to transfer leadership to TS 3e47e633f9784af7b18e39250f7211df
I20260812 19:12:01.315541 20551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599"
dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
mode: GRACEFUL
new_leader_uuid: "c9645294d712440283be14a35f4e4beb"
from {username='slave'} at 127.0.0.1:38884
I20260812 19:12:01.316201 20551 raft_consensus.cc:595] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 LEADER]: Received request to transfer leadership to TS c9645294d712440283be14a35f4e4beb
I20260812 19:12:01.318825 20551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed"
dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
mode: GRACEFUL
new_leader_uuid: "3e47e633f9784af7b18e39250f7211df"
from {username='slave'} at 127.0.0.1:38884
I20260812 19:12:01.319362 20551 raft_consensus.cc:595] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 LEADER]: Received request to transfer leadership to TS 3e47e633f9784af7b18e39250f7211df
I20260812 19:12:01.321940 20551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365"
dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
mode: GRACEFUL
new_leader_uuid: "c9645294d712440283be14a35f4e4beb"
from {username='slave'} at 127.0.0.1:38884
I20260812 19:12:01.322520 20551 raft_consensus.cc:595] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 LEADER]: Received request to transfer leadership to TS c9645294d712440283be14a35f4e4beb
I20260812 19:12:01.323678 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260812 19:12:01.324229 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:12:01.399793 20932 raft_consensus.cc:979] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6: : Instructing follower c9645294d712440283be14a35f4e4beb to start an election
I20260812 19:12:01.400545 20932 raft_consensus.cc:1067] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 LEADER]: Signalling peer c9645294d712440283be14a35f4e4beb to start an election
I20260812 19:12:01.402118 20636 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365"
dest_uuid: "c9645294d712440283be14a35f4e4beb"
from {username='slave'} at 127.0.0.1:39264
I20260812 19:12:01.402798 20636 raft_consensus.cc:482] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:01.403182 20636 raft_consensus.cc:3037] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:12:01.409281 20636 raft_consensus.cc:504] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:12:01.411317 20636 leader_election.cc:288] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 4 election: Requested vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:12:01.412343 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
I20260812 19:12:01.412744 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0edf0240b0a483cbf2315307b984365" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:12:01.413045 20551 raft_consensus.cc:3032] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:12:01.413372 20723 raft_consensus.cc:3037] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:12:01.413406 20551 raft_consensus.cc:729] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:12:01.414337 20551 consensus_queue.cc:260] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:12:01.415529 20551 raft_consensus.cc:3037] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:12:01.419842 20723 raft_consensus.cc:2454] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9645294d712440283be14a35f4e4beb in term 4.
I20260812 19:12:01.421358 20551 raft_consensus.cc:2454] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9645294d712440283be14a35f4e4beb in term 4.
I20260812 19:12:01.421268 20595 leader_election.cc:302] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e47e633f9784af7b18e39250f7211df, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:12:01.422478 20963 raft_consensus.cc:2781] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:12:01.423611 20963 raft_consensus.cc:686] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 4 LEADER]: Becoming Leader. State: Replica: c9645294d712440283be14a35f4e4beb, State: Running, Role: LEADER
I20260812 19:12:01.424566 20963 consensus_queue.cc:237] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:12:01.432375 20423 catalog_manager.cc:5495] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb reported cstate change: term changed from 3 to 4, leader changed from 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129) to c9645294d712440283be14a35f4e4beb (127.19.70.130). New cstate: current_term: 4 leader_uuid: "c9645294d712440283be14a35f4e4beb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:01.589088 20971 raft_consensus.cc:979] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6: : Instructing follower 3e47e633f9784af7b18e39250f7211df to start an election
I20260812 19:12:01.589747 20932 raft_consensus.cc:1067] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Signalling peer 3e47e633f9784af7b18e39250f7211df to start an election
I20260812 19:12:01.591869 20723 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "147c4ad62d7b4714815454a49c2abd7b"
dest_uuid: "3e47e633f9784af7b18e39250f7211df"
from {username='slave'} at 127.0.0.1:55570
I20260812 19:12:01.592514 20723 raft_consensus.cc:482] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:01.592896 20723 raft_consensus.cc:3037] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:01.598711 20723 raft_consensus.cc:504] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:12:01.600962 20723 leader_election.cc:288] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 2 election: Requested vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), c9645294d712440283be14a35f4e4beb (127.19.70.130:39101)
I20260812 19:12:01.602087 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147c4ad62d7b4714815454a49c2abd7b" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
I20260812 19:12:01.602102 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147c4ad62d7b4714815454a49c2abd7b" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:12:01.602805 20551 raft_consensus.cc:3032] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:01.603073 20636 raft_consensus.cc:3037] T 147c4ad62d7b4714815454a49c2abd7b P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:01.603147 20551 raft_consensus.cc:729] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:12:01.603991 20551 consensus_queue.cc:260] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:12:01.605103 20551 raft_consensus.cc:3037] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:01.608774 20636 raft_consensus.cc:2454] T 147c4ad62d7b4714815454a49c2abd7b P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e47e633f9784af7b18e39250f7211df in term 2.
I20260812 19:12:01.610232 20680 leader_election.cc:302] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e47e633f9784af7b18e39250f7211df, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:12:01.610738 20551 raft_consensus.cc:2454] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e47e633f9784af7b18e39250f7211df in term 2.
I20260812 19:12:01.611320 20934 raft_consensus.cc:2781] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:01.612699 20934 raft_consensus.cc:686] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [term 2 LEADER]: Becoming Leader. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Running, Role: LEADER
I20260812 19:12:01.613512 20934 consensus_queue.cc:237] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:12:01.621731 20423 catalog_manager.cc:5495] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df reported cstate change: term changed from 1 to 2, leader changed from 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129) to 3e47e633f9784af7b18e39250f7211df (127.19.70.131). New cstate: current_term: 2 leader_uuid: "3e47e633f9784af7b18e39250f7211df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:01.742273 20962 raft_consensus.cc:979] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6: : Instructing follower c9645294d712440283be14a35f4e4beb to start an election
I20260812 19:12:01.742854 20932 raft_consensus.cc:1067] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 LEADER]: Signalling peer c9645294d712440283be14a35f4e4beb to start an election
I20260812 19:12:01.744454 20636 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599"
dest_uuid: "c9645294d712440283be14a35f4e4beb"
from {username='slave'} at 127.0.0.1:39264
I20260812 19:12:01.744962 20636 raft_consensus.cc:482] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:01.745224 20636 raft_consensus.cc:3037] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:12:01.750321 20636 raft_consensus.cc:504] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:12:01.752369 20636 leader_election.cc:288] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 4 election: Requested vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), 3e47e633f9784af7b18e39250f7211df (127.19.70.131:46591)
I20260812 19:12:01.753274 20723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3e47e633f9784af7b18e39250f7211df"
I20260812 19:12:01.753274 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3856d6355b564cb69521bca5f18c7599" candidate_uuid: "c9645294d712440283be14a35f4e4beb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
I20260812 19:12:01.753925 20723 raft_consensus.cc:3037] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:12:01.754124 20551 raft_consensus.cc:3032] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:12:01.754415 20551 raft_consensus.cc:729] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:12:01.755103 20962 raft_consensus.cc:979] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6: : Instructing follower 3e47e633f9784af7b18e39250f7211df to start an election
I20260812 19:12:01.755681 20962 raft_consensus.cc:1067] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 LEADER]: Signalling peer 3e47e633f9784af7b18e39250f7211df to start an election
I20260812 19:12:01.755162 20551 consensus_queue.cc:260] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:12:01.756798 20551 raft_consensus.cc:3037] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:12:01.757416 20720 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed"
dest_uuid: "3e47e633f9784af7b18e39250f7211df"
from {username='slave'} at 127.0.0.1:55570
I20260812 19:12:01.757859 20720 raft_consensus.cc:482] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:01.758074 20720 raft_consensus.cc:3037] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:12:01.761430 20723 raft_consensus.cc:2454] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9645294d712440283be14a35f4e4beb in term 4.
I20260812 19:12:01.762686 20595 leader_election.cc:302] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e47e633f9784af7b18e39250f7211df, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:12:01.763553 20963 raft_consensus.cc:2781] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:12:01.764112 20963 raft_consensus.cc:686] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 4 LEADER]: Becoming Leader. State: Replica: c9645294d712440283be14a35f4e4beb, State: Running, Role: LEADER
I20260812 19:12:01.764204 20551 raft_consensus.cc:2454] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9645294d712440283be14a35f4e4beb in term 4.
I20260812 19:12:01.764213 20720 raft_consensus.cc:504] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:12:01.765367 20963 consensus_queue.cc:237] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:12:01.767506 20720 leader_election.cc:288] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 4 election: Requested vote from peers 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187), c9645294d712440283be14a35f4e4beb (127.19.70.130:39101)
I20260812 19:12:01.768596 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c9645294d712440283be14a35f4e4beb"
I20260812 19:12:01.771414 20551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ecbe2f6bc54a53b637427ea3ba3fed" candidate_uuid: "3e47e633f9784af7b18e39250f7211df" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2862c0a6eacd49d4a3982d8169e35fa6"
I20260812 19:12:01.772090 20551 raft_consensus.cc:3032] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:12:01.772213 20636 raft_consensus.cc:3037] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:12:01.772467 20551 raft_consensus.cc:729] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2862c0a6eacd49d4a3982d8169e35fa6, State: Running, Role: LEADER
I20260812 19:12:01.773545 20551 consensus_queue.cc:260] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:12:01.773768 20423 catalog_manager.cc:5495] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb reported cstate change: term changed from 3 to 4, leader changed from 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129) to c9645294d712440283be14a35f4e4beb (127.19.70.130). New cstate: current_term: 4 leader_uuid: "c9645294d712440283be14a35f4e4beb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:01.775180 20551 raft_consensus.cc:3037] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:12:01.780215 20636 raft_consensus.cc:2454] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e47e633f9784af7b18e39250f7211df in term 4.
I20260812 19:12:01.781590 20680 leader_election.cc:302] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e47e633f9784af7b18e39250f7211df, c9645294d712440283be14a35f4e4beb; no voters:
I20260812 19:12:01.782795 20934 raft_consensus.cc:2781] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:12:01.783435 20551 raft_consensus.cc:2454] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e47e633f9784af7b18e39250f7211df in term 4.
I20260812 19:12:01.783764 20934 raft_consensus.cc:686] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 4 LEADER]: Becoming Leader. State: Replica: 3e47e633f9784af7b18e39250f7211df, State: Running, Role: LEADER
I20260812 19:12:01.784793 20934 consensus_queue.cc:237] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } }
I20260812 19:12:01.792970 20423 catalog_manager.cc:5495] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df reported cstate change: term changed from 3 to 4, leader changed from 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129) to 3e47e633f9784af7b18e39250f7211df (127.19.70.131). New cstate: current_term: 4 leader_uuid: "3e47e633f9784af7b18e39250f7211df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:01.813818 20551 raft_consensus.cc:1261] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 4 FOLLOWER]: Refusing update from remote peer c9645294d712440283be14a35f4e4beb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:12:01.815336 20963 consensus_queue.cc:1035] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:01.830924 20720 raft_consensus.cc:1261] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 4 FOLLOWER]: Refusing update from remote peer c9645294d712440283be14a35f4e4beb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:12:01.832868 20963 consensus_queue.cc:1035] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:02.007784 20551 raft_consensus.cc:1261] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Refusing update from remote peer 3e47e633f9784af7b18e39250f7211df: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:02.009505 20934 consensus_queue.cc:1035] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [LEADER]: Connected to new peer: Peer: permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:02.023034 20636 raft_consensus.cc:1261] T 147c4ad62d7b4714815454a49c2abd7b P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Refusing update from remote peer 3e47e633f9784af7b18e39250f7211df: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:02.024682 20972 consensus_queue.cc:1035] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:02.202699 20551 raft_consensus.cc:1261] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 4 FOLLOWER]: Refusing update from remote peer c9645294d712440283be14a35f4e4beb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:12:02.204958 20981 consensus_queue.cc:1035] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:02.217959 20720 raft_consensus.cc:1261] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 4 FOLLOWER]: Refusing update from remote peer c9645294d712440283be14a35f4e4beb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:12:02.220392 20981 consensus_queue.cc:1035] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e47e633f9784af7b18e39250f7211df" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:12:02.325176 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:02.325977 20636 raft_consensus.cc:1261] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 4 FOLLOWER]: Refusing update from remote peer 3e47e633f9784af7b18e39250f7211df: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:12:02.327487 20972 consensus_queue.cc:1035] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9645294d712440283be14a35f4e4beb" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:12:02.333920 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:12:02.335050 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:12:02.338775 20551 raft_consensus.cc:1261] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 4 FOLLOWER]: Refusing update from remote peer 3e47e633f9784af7b18e39250f7211df: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:12:02.340783 20985 consensus_queue.cc:1035] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [LEADER]: Connected to new peer: Peer: permanent_uuid: "2862c0a6eacd49d4a3982d8169e35fa6" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:03.336046 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:03.342859 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:12:03.343436 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:12:04.344385 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:04.350026 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:12:04.350582 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:12:05.351504 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:05.357352 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:12:05.357968 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:12:06.359063 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:06.364778 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:12:06.365288 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:12:07.366201 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:07.371649 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:12:07.372177 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:12:08.373108 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:08.379626 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:12:08.380129 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:12:09.381268 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:09.387890 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:12:09.388422 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:12:10.389432 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:10.396219 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260812 19:12:10.396762 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:12:11.432096 19738 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260812 19:12:11.432650 19738 tablet_server.cc:178] TabletServer@127.19.70.129:0 shutting down...
W20260812 19:12:11.462829 20681 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
W20260812 19:12:11.466903 20681 consensus_peers.cc:487] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df -> Peer 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187): Couldn't send request to peer 2862c0a6eacd49d4a3982d8169e35fa6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:11.473950 19738 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260812 19:12:11.474900 19738 tablet_replica.cc:330] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6: stopping tablet replica
I20260812 19:12:11.475790 19738 raft_consensus.cc:2229] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 4 LEADER]: Raft consensus shutting down.
I20260812 19:12:11.476909 19738 raft_consensus.cc:2258] T e3016afd35dd4d4288c4d55f403c268e P 2862c0a6eacd49d4a3982d8169e35fa6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.479233 19738 tablet_replica.cc:330] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6: stopping tablet replica
I20260812 19:12:11.479854 19738 raft_consensus.cc:2229] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.480402 19738 raft_consensus.cc:2258] T b1ecbe2f6bc54a53b637427ea3ba3fed P 2862c0a6eacd49d4a3982d8169e35fa6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.482602 19738 tablet_replica.cc:330] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6: stopping tablet replica
I20260812 19:12:11.483304 19738 raft_consensus.cc:2229] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.483835 19738 raft_consensus.cc:2258] T e0edf0240b0a483cbf2315307b984365 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.485759 19738 tablet_replica.cc:330] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6: stopping tablet replica
I20260812 19:12:11.486333 19738 raft_consensus.cc:2229] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.486908 19738 raft_consensus.cc:2258] T 3856d6355b564cb69521bca5f18c7599 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.488878 19738 tablet_replica.cc:330] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6: stopping tablet replica
I20260812 19:12:11.489411 19738 raft_consensus.cc:2229] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Raft consensus shutting down.
I20260812 19:12:11.490370 19738 raft_consensus.cc:2258] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.492326 19738 tablet_replica.cc:330] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6: stopping tablet replica
I20260812 19:12:11.492868 19738 raft_consensus.cc:2229] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 LEADER]: Raft consensus shutting down.
I20260812 19:12:11.493788 19738 raft_consensus.cc:2258] T 66f8b8ae72814bfbbf087d4847f8cde7 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.495749 19738 tablet_replica.cc:330] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6: stopping tablet replica
I20260812 19:12:11.496284 19738 raft_consensus.cc:2229] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.496757 19738 raft_consensus.cc:2258] T 40269a4340dd4476beeface532a7d1d0 P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.498663 19738 tablet_replica.cc:330] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6: stopping tablet replica
I20260812 19:12:11.499189 19738 raft_consensus.cc:2229] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.499668 19738 raft_consensus.cc:2258] T 0066afe62a194830a36fd1a214ea579a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.502143 19738 tablet_replica.cc:330] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6: stopping tablet replica
I20260812 19:12:11.502884 19738 raft_consensus.cc:2229] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 LEADER]: Raft consensus shutting down.
I20260812 19:12:11.504052 19738 raft_consensus.cc:2258] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 2862c0a6eacd49d4a3982d8169e35fa6 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260812 19:12:11.504614 20681 consensus_peers.cc:487] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df -> Peer 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187): Couldn't send request to peer 2862c0a6eacd49d4a3982d8169e35fa6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:11.506539 19738 tablet_replica.cc:330] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6: stopping tablet replica
I20260812 19:12:11.507294 19738 raft_consensus.cc:2229] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.507962 19738 raft_consensus.cc:2258] T 147c4ad62d7b4714815454a49c2abd7b P 2862c0a6eacd49d4a3982d8169e35fa6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.543475 19738 tablet_server.cc:194] TabletServer@127.19.70.129:0 shutdown complete.
I20260812 19:12:11.565478 19738 tablet_server.cc:178] TabletServer@127.19.70.130:0 shutting down...
W20260812 19:12:11.593825 20680 consensus_peers.cc:487] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df -> Peer c9645294d712440283be14a35f4e4beb (127.19.70.130:39101): Couldn't send request to peer c9645294d712440283be14a35f4e4beb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260812 19:12:11.607224 20595 consensus_peers.cc:487] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb -> Peer 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187): Couldn't send request to peer 2862c0a6eacd49d4a3982d8169e35fa6. Status: Network error: Client connection negotiation failed: client connection to 127.19.70.129:35187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:11.610356 19738 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260812 19:12:11.611253 19738 tablet_replica.cc:330] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb: stopping tablet replica
I20260812 19:12:11.612025 19738 raft_consensus.cc:2229] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [term 2 LEADER]: Raft consensus shutting down.
I20260812 19:12:11.612902 19738 raft_consensus.cc:2258] T 0066afe62a194830a36fd1a214ea579a P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.615036 19738 tablet_replica.cc:330] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb: stopping tablet replica
I20260812 19:12:11.615592 19738 raft_consensus.cc:2229] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 4 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.616073 19738 raft_consensus.cc:2258] T b1ecbe2f6bc54a53b637427ea3ba3fed P c9645294d712440283be14a35f4e4beb [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.617929 19738 tablet_replica.cc:330] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb: stopping tablet replica
I20260812 19:12:11.618516 19738 raft_consensus.cc:2229] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 4 LEADER]: Raft consensus shutting down.
I20260812 19:12:11.619447 19738 raft_consensus.cc:2258] T e0edf0240b0a483cbf2315307b984365 P c9645294d712440283be14a35f4e4beb [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.621201 19738 tablet_replica.cc:330] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb: stopping tablet replica
I20260812 19:12:11.621737 19738 raft_consensus.cc:2229] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 4 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.622363 19738 raft_consensus.cc:2258] T e3016afd35dd4d4288c4d55f403c268e P c9645294d712440283be14a35f4e4beb [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.624666 19738 tablet_replica.cc:330] T 147c4ad62d7b4714815454a49c2abd7b P c9645294d712440283be14a35f4e4beb: stopping tablet replica
I20260812 19:12:11.625226 19738 raft_consensus.cc:2229] T 147c4ad62d7b4714815454a49c2abd7b P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.625751 19738 raft_consensus.cc:2258] T 147c4ad62d7b4714815454a49c2abd7b P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Raft consensus is shut down!
W20260812 19:12:11.627717 20681 consensus_peers.cc:487] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df -> Peer 2862c0a6eacd49d4a3982d8169e35fa6 (127.19.70.129:35187): Couldn't send request to peer 2862c0a6eacd49d4a3982d8169e35fa6. Status: Network error: Client connection negotiation failed: client connection to 127.19.70.129:35187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:11.627852 19738 tablet_replica.cc:330] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb: stopping tablet replica
I20260812 19:12:11.628459 19738 raft_consensus.cc:2229] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 4 LEADER]: Raft consensus shutting down.
I20260812 19:12:11.629362 19738 raft_consensus.cc:2258] T 3856d6355b564cb69521bca5f18c7599 P c9645294d712440283be14a35f4e4beb [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.631400 19738 tablet_replica.cc:330] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P c9645294d712440283be14a35f4e4beb: stopping tablet replica
I20260812 19:12:11.631935 19738 raft_consensus.cc:2229] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.632380 19738 raft_consensus.cc:2258] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.634326 19738 tablet_replica.cc:330] T 66f8b8ae72814bfbbf087d4847f8cde7 P c9645294d712440283be14a35f4e4beb: stopping tablet replica
I20260812 19:12:11.634932 19738 raft_consensus.cc:2229] T 66f8b8ae72814bfbbf087d4847f8cde7 P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.635407 19738 raft_consensus.cc:2258] T 66f8b8ae72814bfbbf087d4847f8cde7 P c9645294d712440283be14a35f4e4beb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.637488 19738 tablet_replica.cc:330] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb: stopping tablet replica
I20260812 19:12:11.638000 19738 raft_consensus.cc:2229] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.638442 19738 raft_consensus.cc:2258] T 40269a4340dd4476beeface532a7d1d0 P c9645294d712440283be14a35f4e4beb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.640280 19738 tablet_replica.cc:330] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb: stopping tablet replica
I20260812 19:12:11.640971 19738 raft_consensus.cc:2229] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.641538 19738 raft_consensus.cc:2258] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P c9645294d712440283be14a35f4e4beb [term 3 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.674033 19738 tablet_server.cc:194] TabletServer@127.19.70.130:0 shutdown complete.
I20260812 19:12:11.695431 19738 tablet_server.cc:178] TabletServer@127.19.70.131:0 shutting down...
I20260812 19:12:11.729247 19738 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260812 19:12:11.730041 19738 tablet_replica.cc:330] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df: stopping tablet replica
I20260812 19:12:11.731030 19738 raft_consensus.cc:2229] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [term 3 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.731594 19738 raft_consensus.cc:2258] T 02bf63ed4d474c0a9b03bde1a2aa2f7a P 3e47e633f9784af7b18e39250f7211df [term 3 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.733701 19738 tablet_replica.cc:330] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df: stopping tablet replica
I20260812 19:12:11.734324 19738 raft_consensus.cc:2229] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 4 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.734890 19738 raft_consensus.cc:2258] T e3016afd35dd4d4288c4d55f403c268e P 3e47e633f9784af7b18e39250f7211df [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.736755 19738 tablet_replica.cc:330] T 66f8b8ae72814bfbbf087d4847f8cde7 P 3e47e633f9784af7b18e39250f7211df: stopping tablet replica
I20260812 19:12:11.737360 19738 raft_consensus.cc:2229] T 66f8b8ae72814bfbbf087d4847f8cde7 P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.737821 19738 raft_consensus.cc:2258] T 66f8b8ae72814bfbbf087d4847f8cde7 P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.739816 19738 tablet_replica.cc:330] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df: stopping tablet replica
I20260812 19:12:11.740415 19738 raft_consensus.cc:2229] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df [term 2 LEADER]: Raft consensus shutting down.
I20260812 19:12:11.741293 19738 raft_consensus.cc:2258] T 40269a4340dd4476beeface532a7d1d0 P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.743324 19738 tablet_replica.cc:330] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df: stopping tablet replica
I20260812 19:12:11.743870 19738 raft_consensus.cc:2229] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 4 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.744410 19738 raft_consensus.cc:2258] T e0edf0240b0a483cbf2315307b984365 P 3e47e633f9784af7b18e39250f7211df [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.746539 19738 tablet_replica.cc:330] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 3e47e633f9784af7b18e39250f7211df: stopping tablet replica
I20260812 19:12:11.747197 19738 raft_consensus.cc:2229] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.747645 19738 raft_consensus.cc:2258] T 6e56a1b8bf6042cc9ac59c1a4bcfa130 P 3e47e633f9784af7b18e39250f7211df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.749430 19738 tablet_replica.cc:330] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df: stopping tablet replica
I20260812 19:12:11.749972 19738 raft_consensus.cc:2229] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 4 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.750468 19738 raft_consensus.cc:2258] T 3856d6355b564cb69521bca5f18c7599 P 3e47e633f9784af7b18e39250f7211df [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.752334 19738 tablet_replica.cc:330] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df: stopping tablet replica
I20260812 19:12:11.752861 19738 raft_consensus.cc:2229] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:11.753317 19738 raft_consensus.cc:2258] T 0066afe62a194830a36fd1a214ea579a P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.755204 19738 tablet_replica.cc:330] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df: stopping tablet replica
I20260812 19:12:11.755718 19738 raft_consensus.cc:2229] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [term 2 LEADER]: Raft consensus shutting down.
I20260812 19:12:11.756562 19738 raft_consensus.cc:2258] T 147c4ad62d7b4714815454a49c2abd7b P 3e47e633f9784af7b18e39250f7211df [term 2 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.758383 19738 tablet_replica.cc:330] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df: stopping tablet replica
I20260812 19:12:11.758901 19738 raft_consensus.cc:2229] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 4 LEADER]: Raft consensus shutting down.
I20260812 19:12:11.759763 19738 raft_consensus.cc:2258] T b1ecbe2f6bc54a53b637427ea3ba3fed P 3e47e633f9784af7b18e39250f7211df [term 4 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.807700 19738 tablet_server.cc:194] TabletServer@127.19.70.131:0 shutdown complete.
I20260812 19:12:11.829023 19738 master.cc:537] Master@127.19.70.190:45491 shutting down...
I20260812 19:12:11.860805 19738 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [term 1 LEADER]: Raft consensus shutting down.
I20260812 19:12:11.861469 19738 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:11.861848 19738 tablet_replica.cc:330] T 00000000000000000000000000000000 P 68688543ce6f465380d67a94afa60e38: stopping tablet replica
I20260812 19:12:11.883028 19738 master.cc:552] Master@127.19.70.190:45491 shutdown complete.
2862c0a6eacd49d4a3982d8169e35fa6 5
3e47e633f9784af7b18e39250f7211df 1
c9645294d712440283be14a35f4e4beb 4
2862c0a6eacd49d4a3982d8169e35fa6 3
3e47e633f9784af7b18e39250f7211df 3
c9645294d712440283be14a35f4e4beb 4
2862c0a6eacd49d4a3982d8169e35fa6 8
3e47e633f9784af7b18e39250f7211df 1
c9645294d712440283be14a35f4e4beb 1
2862c0a6eacd49d4a3982d8169e35fa6 4
3e47e633f9784af7b18e39250f7211df 3
c9645294d712440283be14a35f4e4beb 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29079 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260812 19:12:11.938937 19738 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.70.190:35311
I20260812 19:12:11.940241 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 19:12:11.947378 21010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:12:11.948060 21009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:12:11.948280 21013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:12:11.949713 19738 server_base.cc:1034] running on GCE node
I20260812 19:12:11.951144 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:12:11.951399 19738 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:12:11.951579 19738 hybrid_clock.cc:648] HybridClock initialized: now 1786561931951564 us; error 0 us; skew 500 ppm
I20260812 19:12:11.952152 19738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:12:11.960253 19738 webserver.cc:460] Webserver started at http://127.19.70.190:43943/ using document root <none> and password file <none>
I20260812 19:12:11.960847 19738 fs_manager.cc:359] Metadata directory not provided
I20260812 19:12:11.961077 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:12:11.961414 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:12:11.962875 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/master-0-root/instance:
uuid: "61e1609653694f899be42137777c67c1"
format_stamp: "Formatted at 2026-08-12 19:12:11 on dist-test-slave-pgkr"
I20260812 19:12:11.968103 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 19:12:11.972065 21018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:12:11.973100 19738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 19:12:11.973433 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/master-0-root
uuid: "61e1609653694f899be42137777c67c1"
format_stamp: "Formatted at 2026-08-12 19:12:11 on dist-test-slave-pgkr"
I20260812 19:12:11.973724 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:12:12.016620 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 19:12:12.017967 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:12:12.064554 19738 rpc_server.cc:307] RPC server started. Bound to: 127.19.70.190:35311
I20260812 19:12:12.064700 21077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.70.190:35311 every 8 connection(s)
I20260812 19:12:12.069173 21079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.080855 21079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1: Bootstrap starting.
I20260812 19:12:12.085983 21079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:12.090941 21079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1: No bootstrap required, opened a new log
I20260812 19:12:12.093261 21079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e1609653694f899be42137777c67c1" member_type: VOTER }
I20260812 19:12:12.093972 21079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:12.094286 21079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61e1609653694f899be42137777c67c1, State: Initialized, Role: FOLLOWER
I20260812 19:12:12.095005 21079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e1609653694f899be42137777c67c1" member_type: VOTER }
I20260812 19:12:12.095534 21079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 19:12:12.095796 21079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 19:12:12.096135 21079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:12.100530 21079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e1609653694f899be42137777c67c1" member_type: VOTER }
I20260812 19:12:12.101161 21079 leader_election.cc:302] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61e1609653694f899be42137777c67c1; no voters:
I20260812 19:12:12.102499 21079 leader_election.cc:288] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 19:12:12.103071 21083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:12.104760 21083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [term 1 LEADER]: Becoming Leader. State: Replica: 61e1609653694f899be42137777c67c1, State: Running, Role: LEADER
I20260812 19:12:12.105569 21083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e1609653694f899be42137777c67c1" member_type: VOTER }
I20260812 19:12:12.105791 21079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 19:12:12.110699 21084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61e1609653694f899be42137777c67c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e1609653694f899be42137777c67c1" member_type: VOTER } }
I20260812 19:12:12.111423 21084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [sys.catalog]: This master's current role is: LEADER
I20260812 19:12:12.111402 21085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61e1609653694f899be42137777c67c1. Latest consensus state: current_term: 1 leader_uuid: "61e1609653694f899be42137777c67c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e1609653694f899be42137777c67c1" member_type: VOTER } }
I20260812 19:12:12.112112 21085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1 [sys.catalog]: This master's current role is: LEADER
I20260812 19:12:12.118360 21091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 19:12:12.127271 21091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 19:12:12.128549 19738 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260812 19:12:12.137835 21091 catalog_manager.cc:1347] Generated new cluster ID: 6fe2b93f18b34ab49104849badb7c9c5
I20260812 19:12:12.138183 21091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 19:12:12.159242 21091 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 19:12:12.161099 21091 catalog_manager.cc:1504] Loading token signing keys...
I20260812 19:12:12.173013 21091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1: Generated new TSK 0
I20260812 19:12:12.173786 21091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 19:12:12.195714 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 19:12:12.202168 21101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:12:12.204043 21102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:12:12.205693 21104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:12:12.206110 19738 server_base.cc:1034] running on GCE node
I20260812 19:12:12.207525 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:12:12.207777 19738 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:12:12.208001 19738 hybrid_clock.cc:648] HybridClock initialized: now 1786561932207982 us; error 0 us; skew 500 ppm
I20260812 19:12:12.208585 19738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:12:12.211476 19738 webserver.cc:460] Webserver started at http://127.19.70.129:46609/ using document root <none> and password file <none>
I20260812 19:12:12.212080 19738 fs_manager.cc:359] Metadata directory not provided
I20260812 19:12:12.212313 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:12:12.212616 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:12:12.213951 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-0-root/instance:
uuid: "f741102bfe1441d1b453e5a838649059"
format_stamp: "Formatted at 2026-08-12 19:12:12 on dist-test-slave-pgkr"
I20260812 19:12:12.219208 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 19:12:12.223393 21111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:12:12.224421 19738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 19:12:12.224805 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-0-root
uuid: "f741102bfe1441d1b453e5a838649059"
format_stamp: "Formatted at 2026-08-12 19:12:12 on dist-test-slave-pgkr"
I20260812 19:12:12.225243 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:12:12.238251 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 19:12:12.239692 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:12:12.241364 19738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 19:12:12.244128 19738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:12:12.244416 19738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:12:12.244710 19738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 19:12:12.244867 19738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:12:12.294118 19738 rpc_server.cc:307] RPC server started. Bound to: 127.19.70.129:36551
I20260812 19:12:12.294243 21185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.70.129:36551 every 8 connection(s)
I20260812 19:12:12.299477 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 19:12:12.307720 21191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:12:12.310212 21192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:12:12.312098 21194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:12:12.313251 19738 server_base.cc:1034] running on GCE node
I20260812 19:12:12.314570 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:12:12.315017 19738 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:12:12.315260 19738 hybrid_clock.cc:648] HybridClock initialized: now 1786561932315239 us; error 0 us; skew 500 ppm
I20260812 19:12:12.316073 19738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:12:12.317510 21186 heartbeater.cc:347] Connected to a master server at 127.19.70.190:35311
I20260812 19:12:12.318051 21186 heartbeater.cc:464] Registering TS with master...
I20260812 19:12:12.319060 21186 heartbeater.cc:511] Master 127.19.70.190:35311 requested a full tablet report, sending...
I20260812 19:12:12.319638 19738 webserver.cc:460] Webserver started at http://127.19.70.130:33817/ using document root <none> and password file <none>
I20260812 19:12:12.320344 19738 fs_manager.cc:359] Metadata directory not provided
I20260812 19:12:12.320562 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:12:12.320873 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:12:12.322191 21037 ts_manager.cc:194] Registered new tserver with Master: f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551)
I20260812 19:12:12.322372 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-1-root/instance:
uuid: "d1e1da0b9585460e8c365bf55679d4fa"
format_stamp: "Formatted at 2026-08-12 19:12:12 on dist-test-slave-pgkr"
I20260812 19:12:12.324792 21037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48428
I20260812 19:12:12.328573 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 19:12:12.332834 21199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:12:12.333789 19738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 19:12:12.334168 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-1-root
uuid: "d1e1da0b9585460e8c365bf55679d4fa"
format_stamp: "Formatted at 2026-08-12 19:12:12 on dist-test-slave-pgkr"
I20260812 19:12:12.334529 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:12:12.350266 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 19:12:12.351850 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:12:12.353516 19738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 19:12:12.356333 19738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:12:12.356618 19738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:12:12.356894 19738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 19:12:12.357084 19738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:12:12.401055 19738 rpc_server.cc:307] RPC server started. Bound to: 127.19.70.130:33807
I20260812 19:12:12.401182 21270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.70.130:33807 every 8 connection(s)
I20260812 19:12:12.406355 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 19:12:12.414036 21274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:12:12.417038 21275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:12:12.418213 19738 server_base.cc:1034] running on GCE node
W20260812 19:12:12.419437 21277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:12:12.420581 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:12:12.420836 19738 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:12:12.421042 19738 hybrid_clock.cc:648] HybridClock initialized: now 1786561932421024 us; error 0 us; skew 500 ppm
I20260812 19:12:12.421787 19738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:12:12.422861 21271 heartbeater.cc:347] Connected to a master server at 127.19.70.190:35311
I20260812 19:12:12.423384 21271 heartbeater.cc:464] Registering TS with master...
I20260812 19:12:12.424206 21271 heartbeater.cc:511] Master 127.19.70.190:35311 requested a full tablet report, sending...
I20260812 19:12:12.424889 19738 webserver.cc:460] Webserver started at http://127.19.70.131:45673/ using document root <none> and password file <none>
I20260812 19:12:12.425527 19738 fs_manager.cc:359] Metadata directory not provided
I20260812 19:12:12.425786 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:12:12.426123 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:12:12.426775 21037 ts_manager.cc:194] Registered new tserver with Master: d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:12.427755 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-2-root/instance:
uuid: "93578836968649f6bfcaecc709871412"
format_stamp: "Formatted at 2026-08-12 19:12:12 on dist-test-slave-pgkr"
I20260812 19:12:12.428617 21037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48434
I20260812 19:12:12.433194 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 19:12:12.437487 21284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:12:12.438421 19738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 19:12:12.438813 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-2-root
uuid: "93578836968649f6bfcaecc709871412"
format_stamp: "Formatted at 2026-08-12 19:12:12 on dist-test-slave-pgkr"
I20260812 19:12:12.439105 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:12:12.464270 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 19:12:12.465543 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:12:12.467150 19738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 19:12:12.469596 19738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:12:12.469805 19738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:12:12.470108 19738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 19:12:12.470250 19738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:12:12.514791 19738 rpc_server.cc:307] RPC server started. Bound to: 127.19.70.131:33285
I20260812 19:12:12.514889 21359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.70.131:33285 every 8 connection(s)
I20260812 19:12:12.533677 21360 heartbeater.cc:347] Connected to a master server at 127.19.70.190:35311
I20260812 19:12:12.534400 21360 heartbeater.cc:464] Registering TS with master...
I20260812 19:12:12.535516 21360 heartbeater.cc:511] Master 127.19.70.190:35311 requested a full tablet report, sending...
I20260812 19:12:12.537822 21037 ts_manager.cc:194] Registered new tserver with Master: 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:12.538515 19738 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.020264799s
I20260812 19:12:12.539793 21037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48448
I20260812 19:12:12.541565 19738 test_util.cc:268] Using random seed: 1566293594
I20260812 19:12:12.571617 21037 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48458:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 19:12:12.578521 21037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 19:12:12.801160 21232 tablet_service.cc:1467] Processing CreateTablet for tablet 9ce6040ab9fc41539e7fc09b3f9d05dd (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260812 19:12:12.802806 21232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ce6040ab9fc41539e7fc09b3f9d05dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.812979 21231 tablet_service.cc:1467] Processing CreateTablet for tablet a4c8327f92104d69b0af3f6e7bf28d17 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260812 19:12:12.813591 21229 tablet_service.cc:1467] Processing CreateTablet for tablet eadd5203f7ca4c04b02bd28bea6b591c (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260812 19:12:12.814741 21231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c8327f92104d69b0af3f6e7bf28d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.815433 21229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eadd5203f7ca4c04b02bd28bea6b591c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.815541 21316 tablet_service.cc:1467] Processing CreateTablet for tablet 9ce6040ab9fc41539e7fc09b3f9d05dd (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260812 19:12:12.816958 21316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ce6040ab9fc41539e7fc09b3f9d05dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.818215 21228 tablet_service.cc:1467] Processing CreateTablet for tablet 0e242bf1e9fb45a197a8eaa86bf09f44 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260812 19:12:12.819783 21228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e242bf1e9fb45a197a8eaa86bf09f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.819931 21315 tablet_service.cc:1467] Processing CreateTablet for tablet a4c8327f92104d69b0af3f6e7bf28d17 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260812 19:12:12.821369 21315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c8327f92104d69b0af3f6e7bf28d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.823922 21314 tablet_service.cc:1467] Processing CreateTablet for tablet eadd5203f7ca4c04b02bd28bea6b591c (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260812 19:12:12.825482 21314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eadd5203f7ca4c04b02bd28bea6b591c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.827690 21313 tablet_service.cc:1467] Processing CreateTablet for tablet 0e242bf1e9fb45a197a8eaa86bf09f44 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260812 19:12:12.829293 21313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e242bf1e9fb45a197a8eaa86bf09f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.831951 21311 tablet_service.cc:1467] Processing CreateTablet for tablet 8a915c8829aa4e1fae0110e009af0416 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260812 19:12:12.832628 21226 tablet_service.cc:1467] Processing CreateTablet for tablet 8a915c8829aa4e1fae0110e009af0416 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260812 19:12:12.833603 21311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a915c8829aa4e1fae0110e009af0416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.834554 21226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a915c8829aa4e1fae0110e009af0416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.836097 21310 tablet_service.cc:1467] Processing CreateTablet for tablet e06f074e35674592bdc8360835fd2661 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260812 19:12:12.841292 21309 tablet_service.cc:1467] Processing CreateTablet for tablet 59dcce5cc3a24cc088f6eb1fe3b3a021 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260812 19:12:12.844727 21308 tablet_service.cc:1467] Processing CreateTablet for tablet c6309c277c104b23a58d4824d764d397 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260812 19:12:12.844970 21225 tablet_service.cc:1467] Processing CreateTablet for tablet e06f074e35674592bdc8360835fd2661 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260812 19:12:12.847851 21307 tablet_service.cc:1467] Processing CreateTablet for tablet 85aa3ef35cc045a393fcbb13f15370a7 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260812 19:12:12.849277 21225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06f074e35674592bdc8360835fd2661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.849686 21306 tablet_service.cc:1467] Processing CreateTablet for tablet ae9be4809de24d32bdd888c6f9511d0f (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260812 19:12:12.859087 21310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06f074e35674592bdc8360835fd2661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.859874 21307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85aa3ef35cc045a393fcbb13f15370a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.860986 21224 tablet_service.cc:1467] Processing CreateTablet for tablet 59dcce5cc3a24cc088f6eb1fe3b3a021 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260812 19:12:12.862572 21224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59dcce5cc3a24cc088f6eb1fe3b3a021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.863238 21306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9be4809de24d32bdd888c6f9511d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.863952 21308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6309c277c104b23a58d4824d764d397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.864796 21222 tablet_service.cc:1467] Processing CreateTablet for tablet c6309c277c104b23a58d4824d764d397 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260812 19:12:12.866312 21222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6309c277c104b23a58d4824d764d397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.867669 21309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59dcce5cc3a24cc088f6eb1fe3b3a021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.872898 21221 tablet_service.cc:1467] Processing CreateTablet for tablet 85aa3ef35cc045a393fcbb13f15370a7 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260812 19:12:12.874506 21221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85aa3ef35cc045a393fcbb13f15370a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.880205 21219 tablet_service.cc:1467] Processing CreateTablet for tablet ae9be4809de24d32bdd888c6f9511d0f (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260812 19:12:12.881811 21219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9be4809de24d32bdd888c6f9511d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.929176 21378 tablet_bootstrap.cc:492] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:12.941613 21219 tablet_service.cc:1467] Processing CreateTablet for tablet cab2f57c1cbc44b6976aeb480e73ad2c (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260812 19:12:12.948081 21311 tablet_service.cc:1467] Processing CreateTablet for tablet cab2f57c1cbc44b6976aeb480e73ad2c (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260812 19:12:12.948078 21229 tablet_service.cc:1467] Processing CreateTablet for tablet 65db14a4838d40218947e8bd3420e892 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260812 19:12:12.950280 21311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cab2f57c1cbc44b6976aeb480e73ad2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.952207 21222 tablet_service.cc:1467] Processing CreateTablet for tablet 784d71ae37454b3a9e8023f268428468 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260812 19:12:12.952482 21232 tablet_service.cc:1467] Processing CreateTablet for tablet e6eaa29a6fe94976a6124729431bd8c8 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260812 19:12:12.953872 21219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cab2f57c1cbc44b6976aeb480e73ad2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.970141 21229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65db14a4838d40218947e8bd3420e892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.970845 21310 tablet_service.cc:1467] Processing CreateTablet for tablet aff4931ad1bb4657aa3650a15b43ddcd (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260812 19:12:12.973889 21378 tablet_bootstrap.cc:654] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:12.955370 21231 tablet_service.cc:1467] Processing CreateTablet for tablet a070b29d02d24ef4b36656fcac6d68d9 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260812 19:12:12.959546 21306 tablet_service.cc:1467] Processing CreateTablet for tablet 5d22e3e7455b4eb0806e88ab3f5f4b3b (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260812 19:12:12.960969 21315 tablet_service.cc:1467] Processing CreateTablet for tablet 65db14a4838d40218947e8bd3420e892 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260812 19:12:12.960969 21228 tablet_service.cc:1467] Processing CreateTablet for tablet 5d22e3e7455b4eb0806e88ab3f5f4b3b (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260812 19:12:12.976568 21147 tablet_service.cc:1467] Processing CreateTablet for tablet a4c8327f92104d69b0af3f6e7bf28d17 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260812 19:12:12.977669 21228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d22e3e7455b4eb0806e88ab3f5f4b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.978039 21147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c8327f92104d69b0af3f6e7bf28d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.961660 21221 tablet_service.cc:1467] Processing CreateTablet for tablet aff4931ad1bb4657aa3650a15b43ddcd (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260812 19:12:12.980737 21221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aff4931ad1bb4657aa3650a15b43ddcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.965369 21307 tablet_service.cc:1467] Processing CreateTablet for tablet 784d71ae37454b3a9e8023f268428468 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260812 19:12:12.983211 21222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784d71ae37454b3a9e8023f268428468. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.969103 21314 tablet_service.cc:1467] Processing CreateTablet for tablet a070b29d02d24ef4b36656fcac6d68d9 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260812 19:12:12.984385 21314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a070b29d02d24ef4b36656fcac6d68d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.984481 21146 tablet_service.cc:1467] Processing CreateTablet for tablet eadd5203f7ca4c04b02bd28bea6b591c (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260812 19:12:12.985872 21146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eadd5203f7ca4c04b02bd28bea6b591c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.986980 21306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d22e3e7455b4eb0806e88ab3f5f4b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.987843 21308 tablet_service.cc:1467] Processing CreateTablet for tablet dc64ef202b904bcf809760ddfcd1f140 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260812 19:12:12.989228 21308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc64ef202b904bcf809760ddfcd1f140. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.965993 21309 tablet_service.cc:1467] Processing CreateTablet for tablet e6eaa29a6fe94976a6124729431bd8c8 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260812 19:12:12.990958 21315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65db14a4838d40218947e8bd3420e892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.968587 21224 tablet_service.cc:1467] Processing CreateTablet for tablet dc64ef202b904bcf809760ddfcd1f140 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260812 19:12:12.968897 21148 tablet_service.cc:1467] Processing CreateTablet for tablet 9ce6040ab9fc41539e7fc09b3f9d05dd (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260812 19:12:12.992105 21148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ce6040ab9fc41539e7fc09b3f9d05dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.992540 21145 tablet_service.cc:1467] Processing CreateTablet for tablet 0e242bf1e9fb45a197a8eaa86bf09f44 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260812 19:12:12.993940 21145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e242bf1e9fb45a197a8eaa86bf09f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.995214 21232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6eaa29a6fe94976a6124729431bd8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:12.999302 21231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a070b29d02d24ef4b36656fcac6d68d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.000660 21143 tablet_service.cc:1467] Processing CreateTablet for tablet 8a915c8829aa4e1fae0110e009af0416 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260812 19:12:13.002277 21143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a915c8829aa4e1fae0110e009af0416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.003497 21313 tablet_service.cc:1467] Processing CreateTablet for tablet 83c3d64ee3474fd0b1c7fd42a6cd30d3 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260812 19:12:13.004361 21225 tablet_service.cc:1467] Processing CreateTablet for tablet 83c3d64ee3474fd0b1c7fd42a6cd30d3 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260812 19:12:13.004638 21141 tablet_service.cc:1467] Processing CreateTablet for tablet e06f074e35674592bdc8360835fd2661 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260812 19:12:13.035252 21141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06f074e35674592bdc8360835fd2661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.005568 21224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc64ef202b904bcf809760ddfcd1f140. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.006495 21309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6eaa29a6fe94976a6124729431bd8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.017210 21140 tablet_service.cc:1467] Processing CreateTablet for tablet 59dcce5cc3a24cc088f6eb1fe3b3a021 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260812 19:12:13.020108 21138 tablet_service.cc:1467] Processing CreateTablet for tablet c6309c277c104b23a58d4824d764d397 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260812 19:12:13.043031 21310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aff4931ad1bb4657aa3650a15b43ddcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.020962 21226 tablet_service.cc:1467] Processing CreateTablet for tablet 2117fe1b5b8c42758a313da21e01a17c (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260812 19:12:13.021733 21379 tablet_bootstrap.cc:492] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:13.022696 21137 tablet_service.cc:1467] Processing CreateTablet for tablet 85aa3ef35cc045a393fcbb13f15370a7 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260812 19:12:13.025267 21136 tablet_service.cc:1467] Processing CreateTablet for tablet ae9be4809de24d32bdd888c6f9511d0f (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260812 19:12:13.045836 21138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6309c277c104b23a58d4824d764d397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.059083 21140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59dcce5cc3a24cc088f6eb1fe3b3a021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.046412 21225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c3d64ee3474fd0b1c7fd42a6cd30d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.029850 21316 tablet_service.cc:1467] Processing CreateTablet for tablet 2117fe1b5b8c42758a313da21e01a17c (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260812 19:12:13.072103 21136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9be4809de24d32bdd888c6f9511d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.072913 21137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85aa3ef35cc045a393fcbb13f15370a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.079537 21219 tablet_service.cc:1467] Processing CreateTablet for tablet 4269140f589f4a1fa81fed02bfd8f2bd (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260812 19:12:13.082865 21229 tablet_service.cc:1467] Processing CreateTablet for tablet 9548455d4eee4c74a47bf4b912e5740d (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260812 19:12:13.089929 21226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2117fe1b5b8c42758a313da21e01a17c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.091027 21307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784d71ae37454b3a9e8023f268428468. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.095448 21316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2117fe1b5b8c42758a313da21e01a17c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.098100 21308 tablet_service.cc:1467] Processing CreateTablet for tablet 4269140f589f4a1fa81fed02bfd8f2bd (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260812 19:12:13.099660 21308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4269140f589f4a1fa81fed02bfd8f2bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.101482 21379 tablet_bootstrap.cc:654] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:13.102387 21229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9548455d4eee4c74a47bf4b912e5740d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.111009 21148 tablet_service.cc:1467] Processing CreateTablet for tablet 5d22e3e7455b4eb0806e88ab3f5f4b3b (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260812 19:12:13.112476 21148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d22e3e7455b4eb0806e88ab3f5f4b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.112632 21311 tablet_service.cc:1467] Processing CreateTablet for tablet 9548455d4eee4c74a47bf4b912e5740d (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260812 19:12:13.115782 21219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4269140f589f4a1fa81fed02bfd8f2bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.121668 21381 tablet_bootstrap.cc:492] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
W20260812 19:12:13.123181 21093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260812 19:12:13.127701 21306 tablet_service.cc:1467] Processing CreateTablet for tablet aefbee13b3b74848be07bfcab278a3c8 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260812 19:12:13.128897 21315 tablet_service.cc:1467] Processing CreateTablet for tablet e7e49dcb1cac44ee91df33a2abea1dd0 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260812 19:12:13.138684 21228 tablet_service.cc:1467] Processing CreateTablet for tablet aefbee13b3b74848be07bfcab278a3c8 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260812 19:12:13.146672 21228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aefbee13b3b74848be07bfcab278a3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.153123 21231 tablet_service.cc:1467] Processing CreateTablet for tablet 49361681c5c1421594d10cd52061ec44 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260812 19:12:13.159641 21313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c3d64ee3474fd0b1c7fd42a6cd30d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.160338 21381 tablet_bootstrap.cc:654] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:13.161513 21224 tablet_service.cc:1467] Processing CreateTablet for tablet 41fc49d178354e629e97311498b381ec (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260812 19:12:13.163523 21378 tablet_bootstrap.cc:492] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:13.164119 21378 ts_tablet_manager.cc:1397] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.235s user 0.007s sys 0.023s
I20260812 19:12:13.166213 21311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9548455d4eee4c74a47bf4b912e5740d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.170894 21314 tablet_service.cc:1467] Processing CreateTablet for tablet 41fc49d178354e629e97311498b381ec (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260812 19:12:13.172343 21314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41fc49d178354e629e97311498b381ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.174852 21315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e49dcb1cac44ee91df33a2abea1dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.178774 21231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49361681c5c1421594d10cd52061ec44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.179031 21306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aefbee13b3b74848be07bfcab278a3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.171882 21222 tablet_service.cc:1467] Processing CreateTablet for tablet e7e49dcb1cac44ee91df33a2abea1dd0 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260812 19:12:13.193049 21378 raft_consensus.cc:348] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:13.193945 21378 raft_consensus.cc:374] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:13.194270 21378 raft_consensus.cc:729] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:13.197105 21145 tablet_service.cc:1467] Processing CreateTablet for tablet cab2f57c1cbc44b6976aeb480e73ad2c (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260812 19:12:13.204310 21222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e49dcb1cac44ee91df33a2abea1dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.212006 21145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cab2f57c1cbc44b6976aeb480e73ad2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.214162 21224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41fc49d178354e629e97311498b381ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.216198 21147 tablet_service.cc:1467] Processing CreateTablet for tablet 65db14a4838d40218947e8bd3420e892 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260812 19:12:13.217661 21147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65db14a4838d40218947e8bd3420e892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.219872 21310 tablet_service.cc:1467] Processing CreateTablet for tablet 4c656db261bd45f8b421339ba53dd87f (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260812 19:12:13.221503 21310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c656db261bd45f8b421339ba53dd87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.221318 21138 tablet_service.cc:1467] Processing CreateTablet for tablet e6eaa29a6fe94976a6124729431bd8c8 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260812 19:12:13.222891 21226 tablet_service.cc:1467] Processing CreateTablet for tablet 3f11f84389864fddb8350b0f94829938 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260812 19:12:13.224157 21140 tablet_service.cc:1467] Processing CreateTablet for tablet a070b29d02d24ef4b36656fcac6d68d9 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260812 19:12:13.236982 21232 tablet_service.cc:1467] Processing CreateTablet for tablet 4c656db261bd45f8b421339ba53dd87f (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260812 19:12:13.234133 21378 consensus_queue.cc:260] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:13.231078 21138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6eaa29a6fe94976a6124729431bd8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.261940 21141 tablet_service.cc:1467] Processing CreateTablet for tablet aff4931ad1bb4657aa3650a15b43ddcd (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260812 19:12:13.280640 21309 tablet_service.cc:1467] Processing CreateTablet for tablet 3f11f84389864fddb8350b0f94829938 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260812 19:12:13.278944 21226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f11f84389864fddb8350b0f94829938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.296094 21219 tablet_service.cc:1467] Processing CreateTablet for tablet 4580f6a22878438bb77f8d43fbddf459 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260812 19:12:13.296602 21221 tablet_service.cc:1467] Processing CreateTablet for tablet 2f5090bb13ea4aa792b27b04a7a79688 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260812 19:12:13.294811 21232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c656db261bd45f8b421339ba53dd87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.298975 21221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f5090bb13ea4aa792b27b04a7a79688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.299103 21379 tablet_bootstrap.cc:492] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:13.299569 21379 ts_tablet_manager.cc:1397] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.280s user 0.014s sys 0.009s
I20260812 19:12:13.301079 21228 tablet_service.cc:1467] Processing CreateTablet for tablet c31c6a2accba4d29ba096500b9d1a911 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260812 19:12:13.302579 21379 raft_consensus.cc:348] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:13.304752 21381 tablet_bootstrap.cc:492] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:13.305794 21379 raft_consensus.cc:374] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:13.305949 21381 ts_tablet_manager.cc:1397] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.185s user 0.010s sys 0.006s
I20260812 19:12:13.307406 21379 raft_consensus.cc:729] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:13.281917 21140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a070b29d02d24ef4b36656fcac6d68d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.311542 21141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aff4931ad1bb4657aa3650a15b43ddcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.314677 21225 tablet_service.cc:1467] Processing CreateTablet for tablet 32332933db0a4070bbf6fbabe17e716d (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260812 19:12:13.315227 21146 tablet_service.cc:1467] Processing CreateTablet for tablet dc64ef202b904bcf809760ddfcd1f140 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260812 19:12:13.316676 21146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc64ef202b904bcf809760ddfcd1f140. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.310454 21381 raft_consensus.cc:348] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:13.319607 21381 raft_consensus.cc:374] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:13.319967 21381 raft_consensus.cc:729] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:13.321020 21381 consensus_queue.cc:260] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:13.294843 21309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f11f84389864fddb8350b0f94829938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.302591 21228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31c6a2accba4d29ba096500b9d1a911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.330093 21186 heartbeater.cc:503] Master 127.19.70.190:35311 was elected leader, sending a full tablet report...
I20260812 19:12:13.267406 21143 tablet_service.cc:1467] Processing CreateTablet for tablet 784d71ae37454b3a9e8023f268428468 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260812 19:12:13.322016 21384 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:13.348445 21136 tablet_service.cc:1467] Processing CreateTablet for tablet 83c3d64ee3474fd0b1c7fd42a6cd30d3 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260812 19:12:13.350433 21136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c3d64ee3474fd0b1c7fd42a6cd30d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.350340 21148 tablet_service.cc:1467] Processing CreateTablet for tablet 2117fe1b5b8c42758a313da21e01a17c (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260812 19:12:13.366997 21225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32332933db0a4070bbf6fbabe17e716d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.354576 21384 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:13.392457 21311 tablet_service.cc:1467] Processing CreateTablet for tablet 2f5090bb13ea4aa792b27b04a7a79688 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260812 19:12:13.394043 21311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f5090bb13ea4aa792b27b04a7a79688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.323872 21229 tablet_service.cc:1467] Processing CreateTablet for tablet dfa25b5175f14002a8b5ec8869d7c1d7 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260812 19:12:13.395300 21384 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:13.396512 21143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784d71ae37454b3a9e8023f268428468. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.396456 21314 tablet_service.cc:1467] Processing CreateTablet for tablet 4580f6a22878438bb77f8d43fbddf459 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260812 19:12:13.397958 21314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4580f6a22878438bb77f8d43fbddf459. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.399457 21148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2117fe1b5b8c42758a313da21e01a17c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.403564 21271 heartbeater.cc:503] Master 127.19.70.190:35311 was elected leader, sending a full tablet report...
I20260812 19:12:13.395439 21229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa25b5175f14002a8b5ec8869d7c1d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.341007 21379 consensus_queue.cc:260] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:13.441654 21137 tablet_service.cc:1467] Processing CreateTablet for tablet 4269140f589f4a1fa81fed02bfd8f2bd (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260812 19:12:13.443310 21137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4269140f589f4a1fa81fed02bfd8f2bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.443459 21381 ts_tablet_manager.cc:1428] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.136s user 0.003s sys 0.017s
I20260812 19:12:13.445070 21381 tablet_bootstrap.cc:492] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:13.442062 21360 heartbeater.cc:503] Master 127.19.70.190:35311 was elected leader, sending a full tablet report...
I20260812 19:12:13.449729 21147 tablet_service.cc:1467] Processing CreateTablet for tablet 9548455d4eee4c74a47bf4b912e5740d (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260812 19:12:13.451982 21307 tablet_service.cc:1467] Processing CreateTablet for tablet dfa25b5175f14002a8b5ec8869d7c1d7 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260812 19:12:13.453447 21307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa25b5175f14002a8b5ec8869d7c1d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.464196 21222 tablet_service.cc:1467] Processing CreateTablet for tablet fc50dccc6345495aaec42b51dc30d299 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260812 19:12:13.476670 21226 tablet_service.cc:1467] Processing CreateTablet for tablet ad64c74a7de6449c8152bc0e093a59db (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260812 19:12:13.477185 21306 tablet_service.cc:1467] Processing CreateTablet for tablet 32332933db0a4070bbf6fbabe17e716d (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260812 19:12:13.470139 21219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4580f6a22878438bb77f8d43fbddf459. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.485673 21226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad64c74a7de6449c8152bc0e093a59db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.488379 21313 tablet_service.cc:1467] Processing CreateTablet for tablet c31c6a2accba4d29ba096500b9d1a911 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260812 19:12:13.479650 21306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32332933db0a4070bbf6fbabe17e716d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.495045 21313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31c6a2accba4d29ba096500b9d1a911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.496239 21232 tablet_service.cc:1467] Processing CreateTablet for tablet 5405717c8c644382be14b4f6cd0a3bd7 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260812 19:12:13.497731 21232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5405717c8c644382be14b4f6cd0a3bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.507094 21222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc50dccc6345495aaec42b51dc30d299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.515007 21147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9548455d4eee4c74a47bf4b912e5740d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.515484 21315 tablet_service.cc:1467] Processing CreateTablet for tablet 49361681c5c1421594d10cd52061ec44 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260812 19:12:13.529364 21224 tablet_service.cc:1467] Processing CreateTablet for tablet 66cbc2eec68f4a328fa4cf6c0de63656 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260812 19:12:13.530380 21378 ts_tablet_manager.cc:1428] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.366s user 0.044s sys 0.038s
I20260812 19:12:13.531536 21378 tablet_bootstrap.cc:492] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:13.532469 21316 tablet_service.cc:1467] Processing CreateTablet for tablet ad64c74a7de6449c8152bc0e093a59db (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260812 19:12:13.533953 21316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad64c74a7de6449c8152bc0e093a59db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.546133 21315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49361681c5c1421594d10cd52061ec44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.565304 21386 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:13.566375 21390 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:13.565905 21386 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:13.566999 21145 tablet_service.cc:1467] Processing CreateTablet for tablet aefbee13b3b74848be07bfcab278a3c8 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260812 19:12:13.568826 21145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aefbee13b3b74848be07bfcab278a3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.566999 21390 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:13.533094 21308 tablet_service.cc:1467] Processing CreateTablet for tablet 5405717c8c644382be14b4f6cd0a3bd7 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260812 19:12:13.571183 21308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5405717c8c644382be14b4f6cd0a3bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.586493 21224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66cbc2eec68f4a328fa4cf6c0de63656. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.590809 21379 ts_tablet_manager.cc:1428] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.291s user 0.013s sys 0.006s
I20260812 19:12:13.601958 21140 tablet_service.cc:1467] Processing CreateTablet for tablet 41fc49d178354e629e97311498b381ec (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260812 19:12:13.606979 21310 tablet_service.cc:1467] Processing CreateTablet for tablet 66cbc2eec68f4a328fa4cf6c0de63656 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260812 19:12:13.608450 21310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66cbc2eec68f4a328fa4cf6c0de63656. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.623807 21140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41fc49d178354e629e97311498b381ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.610064 21390 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551)
I20260812 19:12:13.614355 21386 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:13.631212 21381 tablet_bootstrap.cc:654] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:13.653517 21378 tablet_bootstrap.cc:654] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:13.633170 21379 tablet_bootstrap.cc:492] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:13.680258 21231 tablet_service.cc:1467] Processing CreateTablet for tablet 7825e8ab0b2e4d7abbb7dd19c9e3ad69 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260812 19:12:13.701092 21310 tablet_service.cc:1467] Processing CreateTablet for tablet fc50dccc6345495aaec42b51dc30d299 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260812 19:12:13.706691 21136 tablet_service.cc:1467] Processing CreateTablet for tablet e7e49dcb1cac44ee91df33a2abea1dd0 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260812 19:12:13.708572 21219 tablet_service.cc:1467] Processing CreateTablet for tablet 414940a0231d4db9a5cba87d50c5f122 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260812 19:12:13.710911 21310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc50dccc6345495aaec42b51dc30d299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.717522 21221 tablet_service.cc:1467] Processing CreateTablet for tablet 392684a63faf471c902391da6a7b0308 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260812 19:12:13.787700 21229 tablet_service.cc:1467] Processing CreateTablet for tablet 7dfc261e3ffb48ddaa432f57bf7aad22 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260812 19:12:13.782344 21231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7825e8ab0b2e4d7abbb7dd19c9e3ad69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.797663 21226 tablet_service.cc:1467] Processing CreateTablet for tablet b04322c3cc8b46a190cd46147e1c66d9 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260812 19:12:13.784128 21379 tablet_bootstrap.cc:654] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:13.794368 21232 tablet_service.cc:1467] Processing CreateTablet for tablet d6521960ce6040af919a9e227d2211ec (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260812 19:12:13.817003 21221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 392684a63faf471c902391da6a7b0308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.821973 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:13.823177 21232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6521960ce6040af919a9e227d2211ec. 1 dirs total, 0 dirs full, 0 dirs failed
W20260812 19:12:13.824265 21200 leader_election.cc:341] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:13.825984 21226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04322c3cc8b46a190cd46147e1c66d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.827207 21219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414940a0231d4db9a5cba87d50c5f122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.828321 21229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dfc261e3ffb48ddaa432f57bf7aad22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.832520 21314 tablet_service.cc:1467] Processing CreateTablet for tablet 392684a63faf471c902391da6a7b0308 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260812 19:12:13.834184 21314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 392684a63faf471c902391da6a7b0308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.840690 21311 tablet_service.cc:1467] Processing CreateTablet for tablet 7825e8ab0b2e4d7abbb7dd19c9e3ad69 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260812 19:12:13.840893 21221 tablet_service.cc:1467] Processing CreateTablet for tablet 18c9ea4c9e414d1093d28b6c91e19ff0 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260812 19:12:13.842356 21221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c9ea4c9e414d1093d28b6c91e19ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.842463 21378 tablet_bootstrap.cc:492] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:13.843179 21378 ts_tablet_manager.cc:1397] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.312s user 0.016s sys 0.026s
I20260812 19:12:13.841034 21309 tablet_service.cc:1467] Processing CreateTablet for tablet 414940a0231d4db9a5cba87d50c5f122 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260812 19:12:13.848181 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:13.848779 21136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e49dcb1cac44ee91df33a2abea1dd0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260812 19:12:13.850410 21203 leader_election.cc:341] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:13.850944 21203 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:13.852028 21385 raft_consensus.cc:2726] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:13.856472 21148 tablet_service.cc:1467] Processing CreateTablet for tablet 4c656db261bd45f8b421339ba53dd87f (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260812 19:12:13.858084 21148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c656db261bd45f8b421339ba53dd87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.859052 21141 tablet_service.cc:1467] Processing CreateTablet for tablet 49361681c5c1421594d10cd52061ec44 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260812 19:12:13.859753 21310 tablet_service.cc:1467] Processing CreateTablet for tablet 7dfc261e3ffb48ddaa432f57bf7aad22 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260812 19:12:13.860095 21137 tablet_service.cc:1467] Processing CreateTablet for tablet 4580f6a22878438bb77f8d43fbddf459 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260812 19:12:13.860735 21141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49361681c5c1421594d10cd52061ec44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.862707 21138 tablet_service.cc:1467] Processing CreateTablet for tablet 2f5090bb13ea4aa792b27b04a7a79688 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260812 19:12:13.864143 21138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f5090bb13ea4aa792b27b04a7a79688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.867161 21137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4580f6a22878438bb77f8d43fbddf459. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.856629 21146 tablet_service.cc:1467] Processing CreateTablet for tablet 3f11f84389864fddb8350b0f94829938 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260812 19:12:13.872483 21146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f11f84389864fddb8350b0f94829938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.873066 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:13.873811 21329 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:13.875471 21112 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:13.876430 21386 raft_consensus.cc:2781] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:13.876796 21386 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:13.877115 21386 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:13.880419 21381 tablet_bootstrap.cc:492] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:13.880978 21381 ts_tablet_manager.cc:1397] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.436s user 0.032s sys 0.020s
I20260812 19:12:13.800395 21225 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1f5b1c2f184b59b8f452b7f8351692 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260812 19:12:13.886549 21225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1f5b1c2f184b59b8f452b7f8351692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.893646 21313 tablet_service.cc:1467] Processing CreateTablet for tablet d6521960ce6040af919a9e227d2211ec (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260812 19:12:13.894976 21228 tablet_service.cc:1467] Processing CreateTablet for tablet 932fac0d61e94da3a0f9c11ae2f597c3 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260812 19:12:13.895293 21231 tablet_service.cc:1467] Processing CreateTablet for tablet 1c678904ece144c29ee1c70ff4568969 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260812 19:12:13.896413 21228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 932fac0d61e94da3a0f9c11ae2f597c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.897909 21231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c678904ece144c29ee1c70ff4568969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.899915 21386 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:13.900517 21378 raft_consensus.cc:348] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:13.901350 21378 raft_consensus.cc:374] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:13.901687 21378 raft_consensus.cc:729] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:13.904080 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:13.904759 21329 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:13.906162 21147 tablet_service.cc:1467] Processing CreateTablet for tablet dfa25b5175f14002a8b5ec8869d7c1d7 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260812 19:12:13.907686 21147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa25b5175f14002a8b5ec8869d7c1d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.909227 21308 tablet_service.cc:1467] Processing CreateTablet for tablet b04322c3cc8b46a190cd46147e1c66d9 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260812 19:12:13.914921 21381 raft_consensus.cc:348] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:13.915316 21378 consensus_queue.cc:260] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:13.884840 21222 tablet_service.cc:1467] Processing CreateTablet for tablet 0569d19f4a06475c84bdc61b1c5f549f (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260812 19:12:13.941310 21222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0569d19f4a06475c84bdc61b1c5f549f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.945451 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:12:13.947364 21287 leader_election.cc:341] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:13.932391 21378 ts_tablet_manager.cc:1428] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.089s user 0.005s sys 0.001s
I20260812 19:12:13.951114 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:13.953171 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:13.953016 21378 tablet_bootstrap.cc:492] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
W20260812 19:12:13.953260 21114 leader_election.cc:341] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260812 19:12:13.955641 21114 leader_election.cc:341] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:13.957134 21306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 31) took 1023 ms. Trace:
I20260812 19:12:13.957589 21306 rpcz_store.cc:276] 0812 19:12:12.934125 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.476233 (+542108us) service_pool.cc:227] Handling call
0812 19:12:13.478099 (+ 1866us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.478199 (+ 100us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:13.758399 (+280200us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:13.956538 (+198139us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:13.957049 (+ 511us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:13.922995 21379 tablet_bootstrap.cc:492] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:13.974123 21379 ts_tablet_manager.cc:1397] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.354s user 0.011s sys 0.014s
I20260812 19:12:13.977415 21379 raft_consensus.cc:348] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:13.978279 21379 raft_consensus.cc:374] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:13.935649 21329 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:13.916697 21311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7825e8ab0b2e4d7abbb7dd19c9e3ad69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.979017 21143 tablet_service.cc:1467] Processing CreateTablet for tablet 32332933db0a4070bbf6fbabe17e716d (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260812 19:12:13.980497 21143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32332933db0a4070bbf6fbabe17e716d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.980430 21112 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:13.981426 21306 tablet_service.cc:1467] Processing CreateTablet for tablet 18c9ea4c9e414d1093d28b6c91e19ff0 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260812 19:12:13.982590 21309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414940a0231d4db9a5cba87d50c5f122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.984122 21379 raft_consensus.cc:729] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:13.984723 21313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6521960ce6040af919a9e227d2211ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.985687 21310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dfc261e3ffb48ddaa432f57bf7aad22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.915920 21381 raft_consensus.cc:374] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:13.987447 21381 raft_consensus.cc:729] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:13.988548 21381 consensus_queue.cc:260] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:13.991832 21306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c9ea4c9e414d1093d28b6c91e19ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:13.923919 21307 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1f5b1c2f184b59b8f452b7f8351692 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260812 19:12:13.985153 21379 consensus_queue.cc:260] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:13.918758 21386 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:13.997920 21386 raft_consensus.cc:2781] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:13.973073 21315 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 29) took 1039 ms. Trace:
I20260812 19:12:14.001577 21315 rpcz_store.cc:276] 0812 19:12:12.933333 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.431754 (+498421us) service_pool.cc:227] Handling call
0812 19:12:13.536724 (+104970us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.545708 (+ 8984us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:13.741638 (+195930us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:13.972471 (+230833us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:13.973002 (+ 531us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.008555 21219 tablet_service.cc:1467] Processing CreateTablet for tablet 3dcaba8d2e2c42b48d7fa20deee5b73f (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260812 19:12:14.018975 21315 tablet_service.cc:1467] Processing CreateTablet for tablet 0569d19f4a06475c84bdc61b1c5f549f (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260812 19:12:13.998685 21386 raft_consensus.cc:686] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:13.998664 21379 ts_tablet_manager.cc:1428] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.024s user 0.006s sys 0.002s
W20260812 19:12:14.035713 21361 tablet.cc:2369] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 19:12:14.057348 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 77023.2us
I20260812 19:12:14.059172 21386 consensus_queue.cc:237] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.063462 21379 tablet_bootstrap.cc:492] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:14.053354 21219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dcaba8d2e2c42b48d7fa20deee5b73f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.039940 21316 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 33) took 1064 ms. Trace:
I20260812 19:12:14.041249 21145 tablet_service.cc:1467] Processing CreateTablet for tablet c31c6a2accba4d29ba096500b9d1a911 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260812 19:12:14.043265 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:14.045463 21308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04322c3cc8b46a190cd46147e1c66d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.046263 21378 tablet_bootstrap.cc:654] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:14.049651 21224 tablet_service.cc:1467] Processing CreateTablet for tablet 45e03c303e40432e8f5da639f7edff82 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260812 19:12:14.078824 21307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1f5b1c2f184b59b8f452b7f8351692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.079605 21224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e03c303e40432e8f5da639f7edff82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.080045 21287 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:14.081451 21390 raft_consensus.cc:2726] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:14.075848 21316 rpcz_store.cc:276] 0812 19:12:12.975289 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.531415 (+556126us) service_pool.cc:227] Handling call
0812 19:12:13.533402 (+ 1987us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.533511 (+ 109us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:13.809650 (+276139us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.039338 (+229688us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.039867 (+ 529us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.084749 21316 tablet_service.cc:1467] Processing CreateTablet for tablet 1c678904ece144c29ee1c70ff4568969 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260812 19:12:14.086345 21316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c678904ece144c29ee1c70ff4568969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.078401 21145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31c6a2accba4d29ba096500b9d1a911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.094179 21146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 25) took 1037 ms. Trace:
I20260812 19:12:14.096758 21226 tablet_service.cc:1467] Processing CreateTablet for tablet d58350d75f2f4a61b79148bd023ff99c (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260812 19:12:14.105170 21148 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 26) took 1047 ms. Trace:
I20260812 19:12:14.096791 21231 tablet_service.cc:1467] Processing CreateTablet for tablet db5ddd3c989548e5a199029ccf42040e (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260812 19:12:14.052441 21140 tablet_service.cc:1467] Processing CreateTablet for tablet ad64c74a7de6449c8152bc0e093a59db (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260812 19:12:14.116523 21140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad64c74a7de6449c8152bc0e093a59db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.116799 21225 tablet_service.cc:1467] Processing CreateTablet for tablet f2bb026b4ce64ee6beb14a67a87b06af (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260812 19:12:14.111277 21148 rpcz_store.cc:276] 0812 19:12:13.057501 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.855450 (+797949us) service_pool.cc:227] Handling call
0812 19:12:13.857542 (+ 2092us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.857651 (+ 109us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:13.950406 (+ 92755us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.104539 (+154133us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.105092 (+ 553us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.120143 21141 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 29) took 1061 ms. Trace:
I20260812 19:12:14.120586 21141 rpcz_store.cc:276] 0812 19:12:13.058540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.857936 (+799396us) service_pool.cc:227] Handling call
0812 19:12:13.860150 (+ 2214us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.860299 (+ 149us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.048229 (+187930us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.119536 (+ 71307us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.120084 (+ 548us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.123886 21137 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 27) took 1066 ms. Trace:
I20260812 19:12:14.122455 21148 tablet_service.cc:1467] Processing CreateTablet for tablet 5405717c8c644382be14b4f6cd0a3bd7 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260812 19:12:14.124351 21137 rpcz_store.cc:276] 0812 19:12:13.057842 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.859213 (+801371us) service_pool.cc:227] Handling call
0812 19:12:13.860860 (+ 1647us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.860951 (+ 91us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.057790 (+196839us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.123317 (+ 65527us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.123818 (+ 501us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:12:14.125742 21093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260812 19:12:14.126752 21315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0569d19f4a06475c84bdc61b1c5f549f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.128343 21229 tablet_service.cc:1467] Processing CreateTablet for tablet a8161580b52441ab9a17a0e1597f243b (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260812 19:12:14.134104 21232 tablet_service.cc:1467] Processing CreateTablet for tablet 38dbcfe527fe4701863d72b9bb480ccf (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260812 19:12:14.094812 21146 rpcz_store.cc:276] 0812 19:12:13.057202 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.855381 (+798179us) service_pool.cc:227] Handling call
0812 19:12:13.871912 (+ 16531us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.872036 (+ 124us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:13.982986 (+110950us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.093586 (+110600us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.094101 (+ 515us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.138403 21228 tablet_service.cc:1467] Processing CreateTablet for tablet 8c13f33db56241a7938cfe52798e77a2 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260812 19:12:14.141942 21148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5405717c8c644382be14b4f6cd0a3bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.122457 21141 tablet_service.cc:1467] Processing CreateTablet for tablet 66cbc2eec68f4a328fa4cf6c0de63656 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260812 19:12:14.126333 21137 tablet_service.cc:1467] Processing CreateTablet for tablet fc50dccc6345495aaec42b51dc30d299 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260812 19:12:14.148378 21231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5ddd3c989548e5a199029ccf42040e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.149042 21381 ts_tablet_manager.cc:1428] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.268s user 0.010s sys 0.031s
I20260812 19:12:14.155920 21221 tablet_service.cc:1467] Processing CreateTablet for tablet b730f70f25e440e68595cc248b40ee98 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260812 19:12:14.139303 21146 tablet_service.cc:1467] Processing CreateTablet for tablet 392684a63faf471c902391da6a7b0308 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260812 19:12:14.170439 21225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2bb026b4ce64ee6beb14a67a87b06af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.174580 21221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b730f70f25e440e68595cc248b40ee98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.175539 21229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8161580b52441ab9a17a0e1597f243b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.176921 21232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38dbcfe527fe4701863d72b9bb480ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.171078 21378 tablet_bootstrap.cc:492] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:14.177525 21141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66cbc2eec68f4a328fa4cf6c0de63656. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.178035 21378 ts_tablet_manager.cc:1397] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.226s user 0.013s sys 0.011s
I20260812 19:12:14.179054 21381 tablet_bootstrap.cc:492] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:14.178054 21226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58350d75f2f4a61b79148bd023ff99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.183089 21138 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 28) took 1124 ms. Trace:
I20260812 19:12:14.183614 21138 rpcz_store.cc:276] 0812 19:12:13.058201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.861712 (+803511us) service_pool.cc:227] Handling call
0812 19:12:13.863616 (+ 1904us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.863716 (+ 100us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:13.930288 (+ 66572us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.168022 (+237734us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.183013 (+ 14991us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.208110 21228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c13f33db56241a7938cfe52798e77a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.209827 21378 raft_consensus.cc:348] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.210799 21378 raft_consensus.cc:374] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:14.211138 21378 raft_consensus.cc:729] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:14.212450 21379 tablet_bootstrap.cc:654] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:14.174743 21035 catalog_manager.cc:5495] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:14.221588 21308 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 42) took 1046 ms. Trace:
I20260812 19:12:14.222076 21308 rpcz_store.cc:276] 0812 19:12:13.175617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.908225 (+732608us) service_pool.cc:227] Handling call
0812 19:12:13.910151 (+ 1926us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.910274 (+ 123us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.146571 (+236297us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.220960 (+ 74389us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.221515 (+ 555us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.225167 21308 tablet_service.cc:1467] Processing CreateTablet for tablet 932fac0d61e94da3a0f9c11ae2f597c3 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260812 19:12:14.236331 21146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 392684a63faf471c902391da6a7b0308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.241215 21306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 44) took 1041 ms. Trace:
I20260812 19:12:14.241741 21306 rpcz_store.cc:276] 0812 19:12:13.199806 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.980397 (+780591us) service_pool.cc:227] Handling call
0812 19:12:13.982423 (+ 2026us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.982531 (+ 108us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.108355 (+125824us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.240495 (+132140us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.241144 (+ 649us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.242967 21137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc50dccc6345495aaec42b51dc30d299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.247438 21385 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:14.248025 21385 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.249513 21147 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 30) took 1183 ms. Trace:
I20260812 19:12:14.250718 21385 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:14.251950 21314 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 37) took 1141 ms. Trace:
I20260812 19:12:14.252580 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:12:14.254958 21200 leader_election.cc:341] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:14.249965 21147 rpcz_store.cc:276] 0812 19:12:13.086960 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.905104 (+818144us) service_pool.cc:227] Handling call
0812 19:12:13.907046 (+ 1942us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.907172 (+ 126us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.091317 (+184145us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.248888 (+157571us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.249445 (+ 557us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.252446 21314 rpcz_store.cc:276] 0812 19:12:13.110449 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.831423 (+720974us) service_pool.cc:227] Handling call
0812 19:12:13.833598 (+ 2175us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.833726 (+ 128us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.155924 (+322198us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.251385 (+ 95461us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.251888 (+ 503us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.262501 21143 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 31) took 1175 ms. Trace:
I20260812 19:12:14.272914 21143 rpcz_store.cc:276] 0812 19:12:13.087389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.977955 (+890566us) service_pool.cc:227] Handling call
0812 19:12:13.979975 (+ 2020us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.980083 (+ 108us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.211528 (+231445us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.261903 (+ 50375us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.262432 (+ 529us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.263520 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:14.274973 21158 raft_consensus.cc:2454] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:14.275045 21314 tablet_service.cc:1467] Processing CreateTablet for tablet 45e03c303e40432e8f5da639f7edff82 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260812 19:12:14.276693 21203 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:12:14.277971 21385 raft_consensus.cc:2781] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:14.278375 21385 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:14.278848 21385 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:14.283135 21143 tablet_service.cc:1467] Processing CreateTablet for tablet 414940a0231d4db9a5cba87d50c5f122 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260812 19:12:14.285320 21222 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37890 (request call id 45) took 1071 ms. Trace:
I20260812 19:12:14.270025 21381 tablet_bootstrap.cc:654] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:14.259215 21378 consensus_queue.cc:260] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.294329 21147 tablet_service.cc:1467] Processing CreateTablet for tablet 7dfc261e3ffb48ddaa432f57bf7aad22 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260812 19:12:14.186231 21138 tablet_service.cc:1467] Processing CreateTablet for tablet 7825e8ab0b2e4d7abbb7dd19c9e3ad69 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260812 19:12:14.283397 21219 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9f5154fdef497797568cd2e10742be (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260812 19:12:14.264734 21308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 932fac0d61e94da3a0f9c11ae2f597c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.298946 21224 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb4e2cce3884f938f1efd2cbbe1a4a3 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260812 19:12:14.299597 21313 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 41) took 1136 ms. Trace:
I20260812 19:12:14.299965 21313 rpcz_store.cc:276] 0812 19:12:13.162923 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.892585 (+729662us) service_pool.cc:227] Handling call
0812 19:12:13.902467 (+ 9882us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.902585 (+ 118us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.157004 (+254419us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.299004 (+142000us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.299533 (+ 529us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14810}
I20260812 19:12:14.285797 21143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414940a0231d4db9a5cba87d50c5f122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.303658 21310 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 40) took 1153 ms. Trace:
I20260812 19:12:14.264312 21136 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 24) took 1207 ms. Trace:
I20260812 19:12:14.304962 21309 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 39) took 1159 ms. Trace:
I20260812 19:12:14.304364 21136 rpcz_store.cc:276] 0812 19:12:13.056882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.674425 (+617543us) service_pool.cc:227] Handling call
0812 19:12:13.848078 (+173653us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.848302 (+ 224us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:13.969477 (+121175us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.263741 (+294264us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.264245 (+ 504us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":61312}
I20260812 19:12:14.271780 21306 tablet_service.cc:1467] Processing CreateTablet for tablet 3dcaba8d2e2c42b48d7fa20deee5b73f (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260812 19:12:14.290132 21385 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.303347 21313 tablet_service.cc:1467] Processing CreateTablet for tablet db5ddd3c989548e5a199029ccf42040e (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260812 19:12:14.291689 21222 rpcz_store.cc:276] 0812 19:12:13.214257 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.865623 (+651366us) service_pool.cc:227] Handling call
0812 19:12:13.940652 (+ 75029us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.940794 (+ 142us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.107900 (+167106us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.284756 (+176856us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.285253 (+ 497us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.311288 21136 tablet_service.cc:1467] Processing CreateTablet for tablet d6521960ce6040af919a9e227d2211ec (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260812 19:12:14.300319 21224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb4e2cce3884f938f1efd2cbbe1a4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.305202 21314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e03c303e40432e8f5da639f7edff82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.306295 21311 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 38) took 1183 ms. Trace:
I20260812 19:12:14.320706 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:14.320860 21311 rpcz_store.cc:276] 0812 19:12:13.122995 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.839659 (+716664us) service_pool.cc:227] Handling call
0812 19:12:13.841737 (+ 2078us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.841849 (+ 112us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.155484 (+313635us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.298315 (+142831us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.306234 (+ 7919us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29051}
I20260812 19:12:14.321470 21158 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:14.304050 21310 rpcz_store.cc:276] 0812 19:12:13.150190 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.858822 (+708632us) service_pool.cc:227] Handling call
0812 19:12:13.860691 (+ 1869us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.860784 (+ 93us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.123425 (+262641us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.303012 (+179587us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.303595 (+ 583us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.336828 21385 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:14.323189 21306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dcaba8d2e2c42b48d7fa20deee5b73f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.339234 21136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6521960ce6040af919a9e227d2211ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.343832 21138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7825e8ab0b2e4d7abbb7dd19c9e3ad69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.351478 21311 tablet_service.cc:1467] Processing CreateTablet for tablet d58350d75f2f4a61b79148bd023ff99c (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260812 19:12:14.351823 21310 tablet_service.cc:1467] Processing CreateTablet for tablet 38dbcfe527fe4701863d72b9bb480ccf (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260812 19:12:14.353282 21313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5ddd3c989548e5a199029ccf42040e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.305362 21309 rpcz_store.cc:276] 0812 19:12:13.145147 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.840165 (+695018us) service_pool.cc:227] Handling call
0812 19:12:13.844297 (+ 4132us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.844400 (+ 103us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.101509 (+257109us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.304293 (+202784us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.304901 (+ 608us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16281}
I20260812 19:12:14.356498 21309 tablet_service.cc:1467] Processing CreateTablet for tablet f2bb026b4ce64ee6beb14a67a87b06af (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260812 19:12:14.358270 21147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dfc261e3ffb48ddaa432f57bf7aad22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.363047 21307 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 43) took 1181 ms. Trace:
I20260812 19:12:14.363286 21219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9f5154fdef497797568cd2e10742be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.363528 21307 rpcz_store.cc:276] 0812 19:12:13.181301 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:13.922914 (+741613us) service_pool.cc:227] Handling call
0812 19:12:13.996147 (+ 73233us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:13.996299 (+ 152us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.216416 (+220117us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.356664 (+140248us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.362972 (+ 6308us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.365948 21307 tablet_service.cc:1467] Processing CreateTablet for tablet a8161580b52441ab9a17a0e1597f243b (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260812 19:12:14.297914 21145 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 32) took 1210 ms. Trace:
I20260812 19:12:14.367005 21145 rpcz_store.cc:276] 0812 19:12:13.087722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.040210 (+952488us) service_pool.cc:227] Handling call
0812 19:12:14.077886 (+ 37676us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.077989 (+ 103us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.182022 (+104033us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.289850 (+107828us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.296007 (+ 6157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.367997 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:14.369194 21316 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 46) took 1083 ms. Trace:
W20260812 19:12:14.370054 21200 leader_election.cc:341] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:14.369939 21145 tablet_service.cc:1467] Processing CreateTablet for tablet b04322c3cc8b46a190cd46147e1c66d9 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260812 19:12:14.373106 21141 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 35) took 1284 ms. Trace:
I20260812 19:12:14.367412 21307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8161580b52441ab9a17a0e1597f243b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.369391 21158 raft_consensus.cc:2454] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
W20260812 19:12:14.387106 21019 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 48377.7us
I20260812 19:12:14.378585 21148 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 34) took 1290 ms. Trace:
I20260812 19:12:14.393338 21390 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:14.398386 21378 ts_tablet_manager.cc:1428] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.220s user 0.016s sys 0.003s
I20260812 19:12:14.398545 21385 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:14.399092 21385 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.399641 21378 tablet_bootstrap.cc:492] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:14.401326 21385 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:14.392060 21203 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:12:14.394980 21390 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.404078 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:14.405508 21390 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:12:14.405648 21203 leader_election.cc:341] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:14.404559 21410 raft_consensus.cc:2781] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:14.409057 21410 raft_consensus.cc:686] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:14.410092 21410 consensus_queue.cc:237] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.412222 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:14.396512 21309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2bb026b4ce64ee6beb14a67a87b06af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.413694 21137 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 36) took 1324 ms. Trace:
I20260812 19:12:14.428769 21143 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 39) took 1291 ms. Trace:
I20260812 19:12:14.392408 21315 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 45) took 1156 ms. Trace:
I20260812 19:12:14.429209 21143 rpcz_store.cc:276] 0812 19:12:13.137140 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.273953 (+1136813us) service_pool.cc:227] Handling call
0812 19:12:14.285182 (+ 11229us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.285305 (+ 123us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.377940 (+ 92635us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.418234 (+ 40294us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.428670 (+ 10436us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.406730 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:14.432196 21143 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1f5b1c2f184b59b8f452b7f8351692 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260812 19:12:14.382045 21141 rpcz_store.cc:276] 0812 19:12:13.088903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.121552 (+1032649us) service_pool.cc:227] Handling call
0812 19:12:14.176938 (+ 55386us) spinlock_profiling.cc:243] Waited 5.82 ms on lock 0x7b5800001898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52f98 00007f52d6b52ebb 00007f52dfa5f8b9 00007f52de9725d8 00007f52dea322a0 00007f52de9e3869 00007f52d7ff1c44 00007f52d7ff1bd2 00007f52d7ff1b01 00007f52d7ff1a7c
0812 19:12:14.176994 (+ 56us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.177142 (+ 148us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.290996 (+113854us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.372539 (+ 81543us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.373035 (+ 496us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13392768}
I20260812 19:12:14.421236 21379 tablet_bootstrap.cc:492] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:14.435801 21141 tablet_service.cc:1467] Processing CreateTablet for tablet 18c9ea4c9e414d1093d28b6c91e19ff0 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260812 19:12:14.437008 21308 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 47) took 1105 ms. Trace:
I20260812 19:12:14.391897 21148 rpcz_store.cc:276] 0812 19:12:13.088516 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.120621 (+1032105us) service_pool.cc:227] Handling call
0812 19:12:14.141134 (+ 20513us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.141405 (+ 271us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.280953 (+139548us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.377875 (+ 96922us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.378504 (+ 629us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:12:14.433597 21287 leader_election.cc:341] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:14.417809 21145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04322c3cc8b46a190cd46147e1c66d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.439177 21148 tablet_service.cc:1467] Processing CreateTablet for tablet 0569d19f4a06475c84bdc61b1c5f549f (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260812 19:12:14.440625 21148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0569d19f4a06475c84bdc61b1c5f549f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.415016 21146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 37) took 1283 ms. Trace:
W20260812 19:12:14.418359 21200 leader_election.cc:341] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:14.379179 21316 rpcz_store.cc:276] 0812 19:12:13.285741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.083711 (+797970us) service_pool.cc:227] Handling call
0812 19:12:14.085690 (+ 1979us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.085914 (+ 224us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.253339 (+167425us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.357239 (+103900us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.369114 (+ 11875us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.444069 21310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38dbcfe527fe4701863d72b9bb480ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.445325 21378 tablet_bootstrap.cc:654] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:14.443277 21141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c9ea4c9e414d1093d28b6c91e19ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.414667 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:14.431413 21140 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 33) took 1343 ms. Trace:
I20260812 19:12:14.441260 21379 ts_tablet_manager.cc:1397] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.377s user 0.020s sys 0.020s
I20260812 19:12:14.437460 21308 rpcz_store.cc:276] 0812 19:12:13.331777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.224084 (+892307us) service_pool.cc:227] Handling call
0812 19:12:14.226169 (+ 2085us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.226294 (+ 125us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.346374 (+120080us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.436300 (+ 89926us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.436942 (+ 642us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.452939 21311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58350d75f2f4a61b79148bd023ff99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.431063 21315 rpcz_store.cc:276] 0812 19:12:13.236159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.017575 (+781416us) service_pool.cc:227] Handling call
0812 19:12:14.071362 (+ 53787us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x7b5800271898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea54d21 00007f52dea321e3 00007f52de9e3869 00007f52d7ff1c44 00007f52d7ff1bd2 00007f52d7ff1b01
0812 19:12:14.071533 (+ 171us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.071670 (+ 137us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.238188 (+166518us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.362242 (+124054us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.391096 (+ 28854us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35503872}
I20260812 19:12:14.459897 21316 tablet_service.cc:1467] Processing CreateTablet for tablet 8c13f33db56241a7938cfe52798e77a2 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260812 19:12:14.462445 21316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c13f33db56241a7938cfe52798e77a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.460752 21315 tablet_service.cc:1467] Processing CreateTablet for tablet b730f70f25e440e68595cc248b40ee98 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260812 19:12:14.451239 21140 rpcz_store.cc:276] 0812 19:12:13.088139 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.051431 (+963292us) service_pool.cc:227] Handling call
0812 19:12:14.115968 (+ 64537us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.116083 (+ 115us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.283463 (+167380us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.430095 (+146632us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.430804 (+ 709us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.471622 21315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b730f70f25e440e68595cc248b40ee98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.429100 21137 rpcz_store.cc:276] 0812 19:12:13.089253 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.125370 (+1036117us) service_pool.cc:227] Handling call
0812 19:12:14.195638 (+ 70268us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b5800001898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea54d21 00007f52dea321e3 00007f52de9e3869 00007f52d7ff1c44 00007f52d7ff1bd2 00007f52d7ff1b01
0812 19:12:14.195848 (+ 210us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.196006 (+ 158us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.374889 (+178883us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.413142 (+ 38253us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.413636 (+ 494us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47953152}
I20260812 19:12:14.462592 21143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1f5b1c2f184b59b8f452b7f8351692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.443315 21200 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:14.442939 21146 rpcz_store.cc:276] 0812 19:12:13.131186 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.137922 (+1006736us) service_pool.cc:227] Handling call
0812 19:12:14.179485 (+ 41563us) spinlock_profiling.cc:243] Waited 3.61 ms on lock 0x7b5800001898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea54d21 00007f52dea321e3 00007f52de9e3869 00007f52d7ff1c44 00007f52d7ff1bd2 00007f52d7ff1b01
0812 19:12:14.179694 (+ 209us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.179841 (+ 147us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.266444 (+ 86603us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.414365 (+147921us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.414954 (+ 589us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8313088}
I20260812 19:12:14.479255 21385 raft_consensus.cc:2726] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:14.480683 21137 tablet_service.cc:1467] Processing CreateTablet for tablet 932fac0d61e94da3a0f9c11ae2f597c3 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260812 19:12:14.480571 21146 tablet_service.cc:1467] Processing CreateTablet for tablet 1c678904ece144c29ee1c70ff4568969 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260812 19:12:14.482079 21137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 932fac0d61e94da3a0f9c11ae2f597c3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260812 19:12:14.482749 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 64441.6us
W20260812 19:12:14.483784 21287 leader_election.cc:341] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:14.484251 21287 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:14.485430 21390 raft_consensus.cc:2726] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:14.487020 21146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c678904ece144c29ee1c70ff4568969. 1 dirs total, 0 dirs full, 0 dirs failed
W20260812 19:12:14.498217 21114 consensus_peers.cc:487] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 -> Peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Couldn't send request to peer d1e1da0b9585460e8c365bf55679d4fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:14.500901 21140 tablet_service.cc:1467] Processing CreateTablet for tablet 3dcaba8d2e2c42b48d7fa20deee5b73f (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260812 19:12:14.508646 21379 raft_consensus.cc:348] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.509518 21379 raft_consensus.cc:374] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:14.509835 21379 raft_consensus.cc:729] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:14.510672 21379 consensus_queue.cc:260] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.511055 21386 consensus_queue.cc:1035] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:14.515561 21140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dcaba8d2e2c42b48d7fa20deee5b73f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.516304 21308 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9f5154fdef497797568cd2e10742be (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260812 19:12:14.518332 21308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9f5154fdef497797568cd2e10742be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.503620 21306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 48) took 1155 ms. Trace:
I20260812 19:12:14.521919 21138 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 38) took 1390 ms. Trace:
I20260812 19:12:14.524238 21379 ts_tablet_manager.cc:1428] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.071s user 0.006s sys 0.000s
I20260812 19:12:14.527931 21313 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 50) took 1086 ms. Trace:
I20260812 19:12:14.530856 21307 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb4e2cce3884f938f1efd2cbbe1a4a3 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260812 19:12:14.539932 21306 rpcz_store.cc:276] 0812 19:12:13.347433 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.243817 (+896384us) service_pool.cc:227] Handling call
0812 19:12:14.312760 (+ 68943us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.313092 (+ 332us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.388864 (+ 75772us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.470193 (+ 81329us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.502972 (+ 32779us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.541961 21313 rpcz_store.cc:276] 0812 19:12:13.441655 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.302295 (+860640us) service_pool.cc:227] Handling call
0812 19:12:14.352700 (+ 50405us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.352834 (+ 134us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.479980 (+127146us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.527357 (+ 47377us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.527860 (+ 503us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.563771 21381 tablet_bootstrap.cc:492] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:14.573231 21381 ts_tablet_manager.cc:1397] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.394s user 0.021s sys 0.010s
I20260812 19:12:14.575488 21147 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 40) took 1424 ms. Trace:
I20260812 19:12:14.576715 21381 raft_consensus.cc:348] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.577605 21381 raft_consensus.cc:374] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:14.577955 21381 raft_consensus.cc:729] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:14.579694 21379 tablet_bootstrap.cc:492] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:14.540688 21138 rpcz_store.cc:276] 0812 19:12:13.131787 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.184849 (+1053062us) service_pool.cc:227] Handling call
0812 19:12:14.315023 (+130174us) spinlock_profiling.cc:243] Waited 7.88 ms on lock 0x7b5800001898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea54d21 00007f52dea321e3 00007f52de9e3869 00007f52d7ff1c44 00007f52d7ff1bd2 00007f52d7ff1b01
0812 19:12:14.343231 (+ 28208us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.343382 (+ 151us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.454147 (+110765us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.521309 (+ 67162us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.521843 (+ 534us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18130688}
I20260812 19:12:14.584733 21138 tablet_service.cc:1467] Processing CreateTablet for tablet 45e03c303e40432e8f5da639f7edff82 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260812 19:12:14.586292 21138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e03c303e40432e8f5da639f7edff82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.587495 21314 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 49) took 1237 ms. Trace:
I20260812 19:12:14.587556 21378 tablet_bootstrap.cc:492] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:14.576236 21147 rpcz_store.cc:276] 0812 19:12:13.151037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.293376 (+1142339us) service_pool.cc:227] Handling call
0812 19:12:14.357651 (+ 64275us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.357811 (+ 160us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.492779 (+134968us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.564177 (+ 71398us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.575400 (+ 11223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.589252 21307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb4e2cce3884f938f1efd2cbbe1a4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.589819 21310 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 52) took 1037 ms. Trace:
I20260812 19:12:14.599879 21378 ts_tablet_manager.cc:1397] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.200s user 0.022s sys 0.002s
I20260812 19:12:14.600677 21381 consensus_queue.cc:260] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.600983 21147 tablet_service.cc:1467] Processing CreateTablet for tablet db5ddd3c989548e5a199029ccf42040e (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260812 19:12:14.605863 21147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5ddd3c989548e5a199029ccf42040e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.603993 21311 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51536 (request call id 51) took 1051 ms. Trace:
I20260812 19:12:14.606843 21311 rpcz_store.cc:276] 0812 19:12:13.552402 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.328550 (+776148us) service_pool.cc:227] Handling call
0812 19:12:14.359678 (+ 31128us) spinlock_profiling.cc:243] Waited 6.71 ms on lock 0x7b5800271898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea54d21 00007f52dea321e3 00007f52de9e3869 00007f52d7ff1c44 00007f52d7ff1bd2 00007f52d7ff1b01
0812 19:12:14.381857 (+ 22179us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.385496 (+ 3639us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.514427 (+128931us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.603432 (+ 89005us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.603930 (+ 498us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15433088}
I20260812 19:12:14.603905 21310 rpcz_store.cc:276] 0812 19:12:13.552777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.350905 (+798128us) service_pool.cc:227] Handling call
0812 19:12:14.388553 (+ 37648us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.392151 (+ 3598us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.532032 (+139881us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.589419 (+ 57387us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.589770 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12394,"spinlock_wait_cycles":3328}
I20260812 19:12:14.604566 21136 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 41) took 1447 ms. Trace:
I20260812 19:12:14.608354 21381 ts_tablet_manager.cc:1428] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.035s user 0.007s sys 0.000s
I20260812 19:12:14.612486 21379 tablet_bootstrap.cc:654] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:14.612986 21141 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 44) took 1423 ms. Trace:
I20260812 19:12:14.613376 21141 rpcz_store.cc:276] 0812 19:12:13.189482 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.434775 (+1245293us) service_pool.cc:227] Handling call
0812 19:12:14.436731 (+ 1956us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.436837 (+ 106us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.533314 (+ 96477us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.599702 (+ 66388us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.612916 (+ 13214us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":705}
I20260812 19:12:14.614511 21381 tablet_bootstrap.cc:492] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:14.610143 21136 rpcz_store.cc:276] 0812 19:12:13.157449 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.305736 (+1148287us) service_pool.cc:227] Handling call
0812 19:12:14.335108 (+ 29372us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.338815 (+ 3707us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.443385 (+104570us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.604047 (+160662us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.604508 (+ 461us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.616278 21148 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 45) took 1409 ms. Trace:
I20260812 19:12:14.588001 21314 rpcz_store.cc:276] 0812 19:12:13.350674 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.273945 (+923271us) service_pool.cc:227] Handling call
0812 19:12:14.276107 (+ 2162us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.276237 (+ 130us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.485042 (+208805us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.586910 (+101868us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.587431 (+ 521us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12772}
I20260812 19:12:14.625835 21136 tablet_service.cc:1467] Processing CreateTablet for tablet d58350d75f2f4a61b79148bd023ff99c (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260812 19:12:14.628196 21386 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:14.628888 21378 raft_consensus.cc:348] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.628741 21386 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.629632 21378 raft_consensus.cc:374] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:14.629930 21378 raft_consensus.cc:729] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:14.630795 21378 consensus_queue.cc:260] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.631422 21386 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:14.633267 21141 tablet_service.cc:1467] Processing CreateTablet for tablet 38dbcfe527fe4701863d72b9bb480ccf (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260812 19:12:14.633934 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:14.634605 21244 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:14.638219 21114 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:14.643042 21379 tablet_bootstrap.cc:492] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:14.643610 21379 ts_tablet_manager.cc:1397] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.015s
I20260812 19:12:14.643807 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:14.645051 21378 ts_tablet_manager.cc:1428] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.036s user 0.006s sys 0.001s
I20260812 19:12:14.646119 21378 tablet_bootstrap.cc:492] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
W20260812 19:12:14.646400 21112 leader_election.cc:341] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:14.648979 21141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38dbcfe527fe4701863d72b9bb480ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.651163 21143 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 43) took 1473 ms. Trace:
I20260812 19:12:14.651602 21143 rpcz_store.cc:276] 0812 19:12:13.177411 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.431159 (+1253748us) service_pool.cc:227] Handling call
0812 19:12:14.449976 (+ 18817us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.453742 (+ 3766us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.531702 (+ 77960us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.637558 (+105856us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.651090 (+ 13532us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.654738 21140 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 48) took 1271 ms. Trace:
I20260812 19:12:14.653847 21143 tablet_service.cc:1467] Processing CreateTablet for tablet f2bb026b4ce64ee6beb14a67a87b06af (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260812 19:12:14.656186 21140 rpcz_store.cc:276] 0812 19:12:13.383288 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.488065 (+1104777us) service_pool.cc:227] Handling call
0812 19:12:14.514989 (+ 26924us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.515115 (+ 126us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.590816 (+ 75701us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.618153 (+ 27337us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.638412 (+ 20259us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b5800001898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52e82 00007f52dfa5f8b9 00007f52de9725d8 00007f52dea36063 00007f52dea31e2f 00007f52dea325ed 00007f52de9e3869 00007f52d7ff1c44 00007f52d7ff1bd2
0812 19:12:14.654657 (+ 16245us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1044,"spinlock_wait_cycles":3667328}
I20260812 19:12:14.658481 21140 tablet_service.cc:1467] Processing CreateTablet for tablet 8c13f33db56241a7938cfe52798e77a2 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260812 19:12:14.618136 21148 rpcz_store.cc:276] 0812 19:12:13.207219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.438164 (+1230945us) service_pool.cc:227] Handling call
0812 19:12:14.440119 (+ 1955us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.440225 (+ 106us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.551405 (+111180us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.615710 (+ 64305us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.616212 (+ 502us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4814}
I20260812 19:12:14.660555 21148 tablet_service.cc:1467] Processing CreateTablet for tablet a8161580b52441ab9a17a0e1597f243b (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260812 19:12:14.655565 21136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58350d75f2f4a61b79148bd023ff99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.661912 21148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8161580b52441ab9a17a0e1597f243b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.662596 21140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c13f33db56241a7938cfe52798e77a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.663406 21143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2bb026b4ce64ee6beb14a67a87b06af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.661207 21378 tablet_bootstrap.cc:654] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:14.672247 21145 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 42) took 1506 ms. Trace:
I20260812 19:12:14.676741 21381 tablet_bootstrap.cc:654] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:14.678666 21420 raft_consensus.cc:2781] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:14.679205 21420 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:14.679631 21420 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:14.677515 21145 rpcz_store.cc:276] 0812 19:12:13.166032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.368936 (+1202904us) service_pool.cc:227] Handling call
0812 19:12:14.385597 (+ 16661us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.388560 (+ 2963us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.576594 (+188034us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.643316 (+ 66722us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.671274 (+ 27958us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1911}
I20260812 19:12:14.672686 21146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 46) took 1406 ms. Trace:
I20260812 19:12:14.667275 21379 raft_consensus.cc:348] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.688814 21145 tablet_service.cc:1467] Processing CreateTablet for tablet b730f70f25e440e68595cc248b40ee98 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260812 19:12:14.686733 21378 tablet_bootstrap.cc:492] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:14.694777 21378 ts_tablet_manager.cc:1397] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.049s user 0.014s sys 0.000s
I20260812 19:12:14.697850 21378 raft_consensus.cc:348] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.698745 21378 raft_consensus.cc:374] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:14.699102 21378 raft_consensus.cc:729] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:14.700024 21378 consensus_queue.cc:260] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.702153 21378 ts_tablet_manager.cc:1428] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260812 19:12:14.687716 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 49599.6us
I20260812 19:12:14.703305 21378 tablet_bootstrap.cc:492] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:14.710032 21378 tablet_bootstrap.cc:654] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:14.714946 21145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b730f70f25e440e68595cc248b40ee98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.687505 21146 rpcz_store.cc:276] 0812 19:12:13.268374 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.479497 (+1211123us) service_pool.cc:227] Handling call
0812 19:12:14.482150 (+ 2653us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.482272 (+ 122us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.580111 (+ 97839us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.640298 (+ 60187us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.671642 (+ 31344us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.691290 21379 raft_consensus.cc:374] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:14.720242 21379 raft_consensus.cc:729] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:14.719492 21146 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9f5154fdef497797568cd2e10742be (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260812 19:12:14.723758 21138 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 49) took 1339 ms. Trace:
I20260812 19:12:14.725374 21147 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 50) took 1341 ms. Trace:
I20260812 19:12:14.725701 21416 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:14.729988 21137 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 47) took 1443 ms. Trace:
I20260812 19:12:14.726256 21416 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.730499 21137 rpcz_store.cc:276] 0812 19:12:13.286264 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.479887 (+1193623us) service_pool.cc:227] Handling call
0812 19:12:14.481552 (+ 1665us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.481663 (+ 111us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.555199 (+ 73536us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.723750 (+168551us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.729909 (+ 6159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.731686 21036 catalog_manager.cc:5495] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:14.733111 21416 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:14.724757 21379 consensus_queue.cc:260] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.735602 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:14.736418 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:14.736944 21379 ts_tablet_manager.cc:1428] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.093s user 0.009s sys 0.000s
W20260812 19:12:14.737277 21287 leader_election.cc:341] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:14.737978 21379 tablet_bootstrap.cc:492] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:14.739835 21137 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb4e2cce3884f938f1efd2cbbe1a4a3 (DEFAULT_TABLE table=test-workload [id=e08e42e3e09549edabde272cb1e5858d]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260812 19:12:14.730378 21381 tablet_bootstrap.cc:492] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:14.741468 21381 ts_tablet_manager.cc:1397] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.127s user 0.029s sys 0.024s
W20260812 19:12:14.742847 21287 leader_election.cc:341] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:14.743347 21287 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:14.744844 21381 raft_consensus.cc:348] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.745675 21381 raft_consensus.cc:374] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:14.745889 21416 raft_consensus.cc:2726] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:14.746027 21381 raft_consensus.cc:729] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:14.748378 21146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9f5154fdef497797568cd2e10742be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.749163 21137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb4e2cce3884f938f1efd2cbbe1a4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:14.724156 21420 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.725916 21147 rpcz_store.cc:276] 0812 19:12:13.384342 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.599973 (+1215631us) service_pool.cc:227] Handling call
0812 19:12:14.605390 (+ 5417us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.605483 (+ 93us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.631319 (+ 25836us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.724272 (+ 92953us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.725315 (+ 1043us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":654}
I20260812 19:12:14.747773 21381 consensus_queue.cc:260] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.753444 21381 ts_tablet_manager.cc:1428] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.012s user 0.005s sys 0.002s
I20260812 19:12:14.754510 21381 tablet_bootstrap.cc:492] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:14.762044 21381 tablet_bootstrap.cc:654] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:14.762966 21148 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 55) took 1098 ms. Trace:
I20260812 19:12:14.770550 21420 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:14.763527 21148 rpcz_store.cc:276] 0812 19:12:13.664153 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.659590 (+995437us) service_pool.cc:227] Handling call
0812 19:12:14.661419 (+ 1829us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.661523 (+ 104us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.727876 (+ 66353us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.754288 (+ 26412us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.762872 (+ 8584us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.771879 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:14.772627 21329 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:14.725437 21138 rpcz_store.cc:276] 0812 19:12:13.383916 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.583692 (+1199776us) service_pool.cc:227] Handling call
0812 19:12:14.585659 (+ 1967us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.585862 (+ 203us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.656095 (+ 70233us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.712846 (+ 56751us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.723305 (+ 10459us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.774499 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:14.781260 21244 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:14.787462 21143 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 53) took 1232 ms. Trace:
I20260812 19:12:14.787909 21143 rpcz_store.cc:276] 0812 19:12:13.554669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.652730 (+1098061us) service_pool.cc:227] Handling call
0812 19:12:14.657061 (+ 4331us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.657531 (+ 470us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.741704 (+ 84173us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.786896 (+ 45192us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.787391 (+ 495us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.793329 21244 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:14.794715 21114 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:14.795730 21420 raft_consensus.cc:2781] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:14.796446 21420 raft_consensus.cc:686] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:14.796626 21140 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 54) took 1191 ms. Trace:
I20260812 19:12:14.797454 21329 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:14.800683 21145 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 56) took 1118 ms. Trace:
I20260812 19:12:14.801123 21145 rpcz_store.cc:276] 0812 19:12:13.682342 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.681158 (+998816us) service_pool.cc:227] Handling call
0812 19:12:14.714369 (+ 33211us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.714485 (+ 116us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.741664 (+ 27179us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.800097 (+ 58433us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.800608 (+ 511us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.804500 21378 tablet_bootstrap.cc:492] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:14.805145 21378 ts_tablet_manager.cc:1397] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.102s user 0.022s sys 0.005s
I20260812 19:12:14.797013 21140 rpcz_store.cc:276] 0812 19:12:13.605282 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.657497 (+1052215us) service_pool.cc:227] Handling call
0812 19:12:14.659424 (+ 1927us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.659523 (+ 99us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.752563 (+ 93040us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.796091 (+ 43528us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.796556 (+ 465us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.797480 21420 consensus_queue.cc:237] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.808730 21378 raft_consensus.cc:348] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.814522 21378 raft_consensus.cc:374] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:14.814946 21378 raft_consensus.cc:729] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:14.815757 21378 consensus_queue.cc:260] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.818490 21378 ts_tablet_manager.cc:1428] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260812 19:12:14.821966 21378 tablet_bootstrap.cc:492] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:14.828991 21381 tablet_bootstrap.cc:492] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:14.829555 21381 ts_tablet_manager.cc:1397] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.075s user 0.016s sys 0.008s
I20260812 19:12:14.833323 21381 raft_consensus.cc:348] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.834226 21381 raft_consensus.cc:374] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:14.834544 21381 raft_consensus.cc:729] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:14.830255 21379 tablet_bootstrap.cc:654] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:14.836997 21146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 57) took 1154 ms. Trace:
I20260812 19:12:14.837586 21146 rpcz_store.cc:276] 0812 19:12:13.682723 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.718458 (+1035735us) service_pool.cc:227] Handling call
0812 19:12:14.723571 (+ 5113us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.726482 (+ 2911us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.804422 (+ 77940us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.836447 (+ 32025us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.836934 (+ 487us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.838587 21381 consensus_queue.cc:260] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.841840 21381 ts_tablet_manager.cc:1428] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260812 19:12:14.843142 21381 tablet_bootstrap.cc:492] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:14.844599 21035 catalog_manager.cc:5495] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:14.848246 21136 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 51) took 1406 ms. Trace:
I20260812 19:12:14.848853 21136 rpcz_store.cc:276] 0812 19:12:13.441655 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.616567 (+1174912us) service_pool.cc:227] Handling call
0812 19:12:14.635975 (+ 19408us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.654985 (+ 19010us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.804203 (+149218us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.847514 (+ 43311us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.848164 (+ 650us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.850266 21410 consensus_queue.cc:1035] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:14.851881 21137 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 58) took 1098 ms. Trace:
I20260812 19:12:14.852782 21381 tablet_bootstrap.cc:654] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:14.852500 21137 rpcz_store.cc:276] 0812 19:12:13.753214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.738119 (+984905us) service_pool.cc:227] Handling call
0812 19:12:14.740752 (+ 2633us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.740852 (+ 100us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.803403 (+ 62551us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.851023 (+ 47620us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.851794 (+ 771us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:12:14.860731 21378 tablet_bootstrap.cc:654] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:14.870069 21141 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54786 (request call id 52) took 1318 ms. Trace:
I20260812 19:12:14.871850 21141 rpcz_store.cc:276] 0812 19:12:13.551503 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:14.621697 (+1070194us) service_pool.cc:227] Handling call
0812 19:12:14.638899 (+ 17202us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b5800001898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea54d21 00007f52dea321e3 00007f52de9e3869 00007f52d7ff1c44 00007f52d7ff1bd2 00007f52d7ff1b01
0812 19:12:14.648247 (+ 9348us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0812 19:12:14.648367 (+ 120us) ts_tablet_manager.cc:676] Creating new metadata...
0812 19:12:14.809197 (+160830us) tablet_metadata.cc:641] Metadata flushed
0812 19:12:14.866247 (+ 57050us) tablet_replica.cc:205] Creating consensus instance
0812 19:12:14.869993 (+ 3746us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4073728}
I20260812 19:12:14.875340 21379 tablet_bootstrap.cc:492] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:14.876157 21379 ts_tablet_manager.cc:1397] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.138s user 0.031s sys 0.011s
I20260812 19:12:14.881919 21379 raft_consensus.cc:348] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.883517 21379 raft_consensus.cc:374] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:14.884171 21379 raft_consensus.cc:729] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:14.885607 21379 consensus_queue.cc:260] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.889611 21379 ts_tablet_manager.cc:1428] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.013s user 0.007s sys 0.003s
I20260812 19:12:14.895538 21420 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:14.896489 21420 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.902912 21379 tablet_bootstrap.cc:492] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
W20260812 19:12:14.903934 21200 consensus_peers.cc:487] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa -> Peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Couldn't send request to peer 93578836968649f6bfcaecc709871412. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:14.904081 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:14.904422 21420 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:12:14.906119 21114 leader_election.cc:341] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:14.906986 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:14.908124 21329 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:14.912089 21112 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:14.913654 21420 raft_consensus.cc:2781] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:14.914300 21420 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:14.914886 21420 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:14.933251 21381 tablet_bootstrap.cc:492] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:14.934062 21381 ts_tablet_manager.cc:1397] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.091s user 0.012s sys 0.012s
I20260812 19:12:14.937055 21410 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:14.937733 21410 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.940229 21420 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.941417 21410 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:14.940328 21379 tablet_bootstrap.cc:654] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:14.942409 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:14.943329 21157 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:14.943759 21420 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:14.944931 21381 raft_consensus.cc:348] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.945477 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:14.946558 21381 raft_consensus.cc:374] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:14.947265 21381 raft_consensus.cc:729] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
W20260812 19:12:14.948056 21114 leader_election.cc:341] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:14.948284 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:14.948457 21381 consensus_queue.cc:260] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.946938 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:12:14.951077 21200 leader_election.cc:341] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:14.951282 21328 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:14.952888 21381 ts_tablet_manager.cc:1428] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.018s user 0.005s sys 0.009s
I20260812 19:12:14.952445 21203 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:12:14.954043 21410 raft_consensus.cc:2781] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:14.954674 21410 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:14.955271 21410 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:14.955116 21420 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:14.956293 21420 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.961853 21385 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:14.963091 21385 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.968372 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:14.970747 21378 tablet_bootstrap.cc:492] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
W20260812 19:12:14.971174 21114 leader_election.cc:341] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:14.971544 21378 ts_tablet_manager.cc:1397] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.151s user 0.031s sys 0.026s
I20260812 19:12:14.971547 21381 tablet_bootstrap.cc:492] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:14.976464 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:14.977438 21378 raft_consensus.cc:348] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.979213 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:14.983788 21378 raft_consensus.cc:374] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:14.984406 21378 raft_consensus.cc:729] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:14.989876 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:14.983062 21385 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:12:14.991987 21112 leader_election.cc:341] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:14.987214 21420 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:14.992875 21112 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:14.994406 21420 raft_consensus.cc:2726] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:14.985013 21203 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:14.996222 21385 raft_consensus.cc:2726] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:14.989712 21378 consensus_queue.cc:260] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:14.999701 21381 tablet_bootstrap.cc:654] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.001864 21378 ts_tablet_manager.cc:1428] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.030s user 0.000s sys 0.012s
I20260812 19:12:14.991693 21410 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.005121 21410 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:15.006531 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:15.007452 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:15.007694 21157 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
W20260812 19:12:15.010070 21200 leader_election.cc:341] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:15.017896 21378 tablet_bootstrap.cc:492] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:15.018196 21328 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:15.023488 21112 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:15.025086 21420 raft_consensus.cc:2781] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:15.025869 21381 tablet_bootstrap.cc:492] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:15.025942 21420 raft_consensus.cc:686] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:15.026824 21381 ts_tablet_manager.cc:1397] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.056s user 0.021s sys 0.010s
I20260812 19:12:15.027545 21420 consensus_queue.cc:237] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.028388 21379 tablet_bootstrap.cc:492] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:15.029206 21379 ts_tablet_manager.cc:1397] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.127s user 0.029s sys 0.064s
I20260812 19:12:15.032096 21381 raft_consensus.cc:348] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.033641 21381 raft_consensus.cc:374] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:15.034212 21381 raft_consensus.cc:729] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:15.034934 21379 raft_consensus.cc:348] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.036403 21379 raft_consensus.cc:374] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:15.035903 21381 consensus_queue.cc:260] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.037127 21379 raft_consensus.cc:729] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:15.038741 21379 consensus_queue.cc:260] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.040828 21381 ts_tablet_manager.cc:1428] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.014s user 0.005s sys 0.007s
I20260812 19:12:15.042285 21381 tablet_bootstrap.cc:492] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:15.044356 21379 ts_tablet_manager.cc:1428] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.015s user 0.006s sys 0.008s
I20260812 19:12:15.046233 21379 tablet_bootstrap.cc:492] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:15.046383 21244 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.049327 21378 tablet_bootstrap.cc:654] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.051359 21385 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:15.052590 21385 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.054366 21381 tablet_bootstrap.cc:654] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.058367 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:15.059574 21385 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:15.059677 21328 raft_consensus.cc:2454] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:15.060775 21379 tablet_bootstrap.cc:654] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.060765 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:15.062238 21200 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:15.064028 21385 raft_consensus.cc:2781] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:15.065220 21385 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:15.066088 21385 raft_consensus.cc:3037] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
W20260812 19:12:15.063952 21203 leader_election.cc:341] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:15.076397 21410 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:15.069272 21157 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:15.077595 21410 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.080744 21203 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:12:15.084776 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:15.086122 21410 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:15.086372 21328 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:15.089994 21410 raft_consensus.cc:2781] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:15.092322 21200 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:15.100679 21379 tablet_bootstrap.cc:492] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:15.101629 21379 ts_tablet_manager.cc:1397] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.056s user 0.013s sys 0.013s
I20260812 19:12:15.110577 21036 catalog_manager.cc:5495] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:15.115450 21379 raft_consensus.cc:348] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.116134 21385 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.118305 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:15.117632 21379 raft_consensus.cc:374] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:15.117156 21410 raft_consensus.cc:686] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:12:15.123469 21203 leader_election.cc:341] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:15.123497 21410 consensus_queue.cc:237] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.128580 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:15.130506 21381 tablet_bootstrap.cc:492] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:15.131446 21381 ts_tablet_manager.cc:1397] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.090s user 0.015s sys 0.014s
W20260812 19:12:15.131330 21203 leader_election.cc:341] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:15.130959 21379 raft_consensus.cc:729] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:15.133315 21385 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:15.136735 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:15.139436 21328 raft_consensus.cc:3037] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.138375 21416 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:15.137346 21443 raft_consensus.cc:2781] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:15.142910 21443 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:15.143726 21443 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.142988 21416 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.144192 21378 tablet_bootstrap.cc:492] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:15.144831 21378 ts_tablet_manager.cc:1397] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.127s user 0.008s sys 0.019s
I20260812 19:12:15.141997 21381 raft_consensus.cc:348] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.147521 21381 raft_consensus.cc:374] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:15.147890 21381 raft_consensus.cc:729] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:15.148900 21381 consensus_queue.cc:260] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.153218 21416 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:15.153481 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:15.159821 21157 raft_consensus.cc:2454] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:15.155604 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:15.161929 21243 raft_consensus.cc:2454] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:15.134459 21379 consensus_queue.cc:260] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.163481 21287 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:15.157109 21381 ts_tablet_manager.cc:1428] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.025s user 0.010s sys 0.000s
W20260812 19:12:15.158838 21093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260812 19:12:15.167055 21381 tablet_bootstrap.cc:492] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:15.170993 21390 raft_consensus.cc:2781] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:15.175081 21390 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:15.175717 21390 raft_consensus.cc:3037] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.182911 21379 ts_tablet_manager.cc:1428] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.081s user 0.006s sys 0.008s
I20260812 19:12:15.184837 21379 tablet_bootstrap.cc:492] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:15.186676 21381 tablet_bootstrap.cc:654] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.186901 21328 raft_consensus.cc:2454] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:15.189294 21200 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:15.191303 21378 raft_consensus.cc:348] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.192848 21378 raft_consensus.cc:374] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:15.193212 21378 raft_consensus.cc:729] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:15.191949 21443 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.193984 21378 consensus_queue.cc:260] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.202986 21378 ts_tablet_manager.cc:1428] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.058s user 0.006s sys 0.005s
I20260812 19:12:15.204138 21378 tablet_bootstrap.cc:492] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:15.192687 21410 raft_consensus.cc:2781] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:15.208087 21410 raft_consensus.cc:686] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:15.209991 21410 consensus_queue.cc:237] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.212224 21379 tablet_bootstrap.cc:654] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.216209 21443 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:15.216040 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:15.216791 21381 tablet_bootstrap.cc:492] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:15.220211 21390 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.224392 21381 ts_tablet_manager.cc:1397] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.058s user 0.015s sys 0.017s
I20260812 19:12:15.223150 21378 tablet_bootstrap.cc:654] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.216106 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:15.227180 21390 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:15.229020 21381 raft_consensus.cc:348] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.230006 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:12:15.226428 21203 leader_election.cc:341] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:15.230978 21157 raft_consensus.cc:3037] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.235873 21328 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.229956 21381 raft_consensus.cc:374] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:15.231026 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:15.237346 21381 raft_consensus.cc:729] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:15.238109 21244 raft_consensus.cc:3037] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.238801 21381 consensus_queue.cc:260] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.260350 21381 ts_tablet_manager.cc:1428] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.036s user 0.004s sys 0.006s
I20260812 19:12:15.261545 21381 tablet_bootstrap.cc:492] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:15.262763 21244 raft_consensus.cc:2454] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:15.265012 21287 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:15.266553 21390 raft_consensus.cc:2781] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:15.267798 21390 raft_consensus.cc:686] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:15.269332 21390 consensus_queue.cc:237] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.277357 21328 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:15.279322 21200 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:15.281243 21410 raft_consensus.cc:2781] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:15.281796 21157 raft_consensus.cc:2454] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:15.282097 21410 raft_consensus.cc:686] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:15.288311 21379 tablet_bootstrap.cc:492] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:15.283757 21410 consensus_queue.cc:237] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.293011 21420 consensus_queue.cc:1035] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:15.289110 21379 ts_tablet_manager.cc:1397] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.105s user 0.024s sys 0.029s
I20260812 19:12:15.295976 21381 tablet_bootstrap.cc:654] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.304018 21378 tablet_bootstrap.cc:492] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:15.304653 21378 ts_tablet_manager.cc:1397] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.101s user 0.021s sys 0.027s
I20260812 19:12:15.294266 21036 catalog_manager.cc:5495] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:15.308133 21379 raft_consensus.cc:348] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.309410 21379 raft_consensus.cc:374] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:15.309891 21379 raft_consensus.cc:729] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:15.310933 21379 consensus_queue.cc:260] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.310130 21036 catalog_manager.cc:5495] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:15.308763 21378 raft_consensus.cc:348] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.314538 21378 raft_consensus.cc:374] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:15.315004 21378 raft_consensus.cc:729] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:15.316226 21378 consensus_queue.cc:260] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.318079 21379 ts_tablet_manager.cc:1428] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.021s user 0.010s sys 0.000s
I20260812 19:12:15.319113 21379 tablet_bootstrap.cc:492] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:15.319367 21443 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:15.320253 21378 ts_tablet_manager.cc:1428] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.015s user 0.009s sys 0.000s
I20260812 19:12:15.320640 21443 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.321414 21378 tablet_bootstrap.cc:492] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:15.322291 21035 catalog_manager.cc:5495] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:15.325398 21381 tablet_bootstrap.cc:492] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:15.325893 21381 ts_tablet_manager.cc:1397] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.014s
I20260812 19:12:15.334308 21379 tablet_bootstrap.cc:654] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.334308 21378 tablet_bootstrap.cc:654] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.337112 21443 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:15.338171 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:15.340737 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:15.344074 21328 raft_consensus.cc:2379] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 1: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:12:15.348138 21200 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:15.349694 21443 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.401376 21378 tablet_bootstrap.cc:492] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:15.400952 21381 raft_consensus.cc:348] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.401574 21379 tablet_bootstrap.cc:492] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:15.401799 21381 raft_consensus.cc:374] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:15.401886 21378 ts_tablet_manager.cc:1397] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.081s user 0.026s sys 0.022s
I20260812 19:12:15.402014 21379 ts_tablet_manager.cc:1397] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.083s user 0.018s sys 0.012s
I20260812 19:12:15.405560 21379 raft_consensus.cc:348] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.405778 21378 raft_consensus.cc:348] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.406427 21379 raft_consensus.cc:374] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:15.406464 21378 raft_consensus.cc:374] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:15.406847 21379 raft_consensus.cc:729] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:15.406965 21378 raft_consensus.cc:729] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:15.407325 21443 raft_consensus.cc:2726] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:15.407686 21379 consensus_queue.cc:260] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.407903 21378 consensus_queue.cc:260] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.410341 21379 ts_tablet_manager.cc:1428] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260812 19:12:15.411525 21379 tablet_bootstrap.cc:492] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:15.413635 21390 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:15.414417 21390 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.420083 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:15.421087 21379 tablet_bootstrap.cc:654] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.421201 21158 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:15.423833 21287 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:15.425733 21416 raft_consensus.cc:2781] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:15.426481 21416 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:15.427232 21416 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.431324 21420 consensus_queue.cc:1035] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:15.432904 21390 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:15.438743 21381 raft_consensus.cc:729] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:15.438974 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:15.449653 21378 ts_tablet_manager.cc:1428] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.047s user 0.001s sys 0.005s
I20260812 19:12:15.452378 21244 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:15.452253 21381 consensus_queue.cc:260] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.456926 21462 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:15.459908 21378 tablet_bootstrap.cc:492] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:15.460287 21381 ts_tablet_manager.cc:1428] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.134s user 0.011s sys 0.007s
I20260812 19:12:15.466320 21381 tablet_bootstrap.cc:492] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:15.471400 21416 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.473869 21379 tablet_bootstrap.cc:492] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:15.475571 21416 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:15.476708 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:15.477423 21158 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.475593 21379 ts_tablet_manager.cc:1397] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.064s user 0.025s sys 0.032s
I20260812 19:12:15.484905 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:15.486287 21244 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.487681 21036 catalog_manager.cc:5495] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:15.467182 21462 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.500072 21462 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:15.501636 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:15.502171 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:15.503311 21329 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:15.503042 21243 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:15.504773 21244 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:15.505144 21112 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:15.494588 21158 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:15.507634 21287 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:15.509438 21416 raft_consensus.cc:2781] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:15.510519 21416 raft_consensus.cc:686] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:15.514591 21416 consensus_queue.cc:237] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.526749 21379 raft_consensus.cc:348] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.528093 21379 raft_consensus.cc:374] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:15.528483 21379 raft_consensus.cc:729] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:15.529412 21379 consensus_queue.cc:260] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.531796 21419 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:15.532621 21419 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.538909 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:15.512068 21462 raft_consensus.cc:2781] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:15.546950 21379 ts_tablet_manager.cc:1428] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.069s user 0.010s sys 0.001s
I20260812 19:12:15.549055 21379 tablet_bootstrap.cc:492] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:15.554126 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:15.555318 21329 raft_consensus.cc:2379] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 1: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:12:15.556974 21112 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:15.558101 21420 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.559201 21443 consensus_queue.cc:1035] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:15.561789 21381 tablet_bootstrap.cc:654] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.545979 21462 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:15.563385 21462 raft_consensus.cc:3037] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.579144 21379 tablet_bootstrap.cc:654] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.583246 21464 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:15.584354 21420 raft_consensus.cc:2726] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:15.587845 21420 consensus_queue.cc:1035] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260812 19:12:15.563629 21419 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:15.591301 21378 tablet_bootstrap.cc:654] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.584074 21464 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.612396 21465 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:15.612527 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:15.613533 21465 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.622510 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:15.622663 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:15.623486 21465 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:12:15.624522 21114 leader_election.cc:341] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:15.625595 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:15.629642 21287 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:15.630160 21464 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:15.641717 21416 raft_consensus.cc:2726] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260812 19:12:15.639735 21112 leader_election.cc:341] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:15.644852 21112 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:15.645709 21419 raft_consensus.cc:2726] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:15.658947 21329 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.665588 21419 consensus_queue.cc:1035] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:15.671880 21379 tablet_bootstrap.cc:492] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:15.672480 21379 ts_tablet_manager.cc:1397] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.124s user 0.025s sys 0.007s
I20260812 19:12:15.676582 21379 raft_consensus.cc:348] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.677486 21379 raft_consensus.cc:374] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:15.677886 21379 raft_consensus.cc:729] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:15.676602 21462 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.683782 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:15.684552 21462 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:15.684731 21243 raft_consensus.cc:3037] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.685379 21379 consensus_queue.cc:260] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.688349 21379 ts_tablet_manager.cc:1428] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.016s user 0.009s sys 0.000s
I20260812 19:12:15.687908 21443 consensus_queue.cc:1035] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:15.689441 21379 tablet_bootstrap.cc:492] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:15.691486 21381 tablet_bootstrap.cc:492] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:15.692139 21381 ts_tablet_manager.cc:1397] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.226s user 0.013s sys 0.050s
I20260812 19:12:15.687314 21036 catalog_manager.cc:5495] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:15.702870 21381 raft_consensus.cc:348] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.703459 21381 raft_consensus.cc:374] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:15.703722 21381 raft_consensus.cc:729] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:15.704272 21381 consensus_queue.cc:260] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.702044 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:15.711090 21378 tablet_bootstrap.cc:492] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:15.747134 21378 ts_tablet_manager.cc:1397] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.287s user 0.010s sys 0.027s
I20260812 19:12:15.748126 21381 ts_tablet_manager.cc:1428] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.056s user 0.019s sys 0.019s
I20260812 19:12:15.749594 21381 tablet_bootstrap.cc:492] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:15.752519 21378 raft_consensus.cc:348] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.754786 21378 raft_consensus.cc:374] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:15.755261 21378 raft_consensus.cc:729] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:15.756240 21378 consensus_queue.cc:260] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.758364 21378 ts_tablet_manager.cc:1428] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260812 19:12:15.760388 21378 tablet_bootstrap.cc:492] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:15.762687 21379 tablet_bootstrap.cc:654] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.753731 21443 consensus_queue.cc:1035] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:15.753458 21329 raft_consensus.cc:3037] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.777230 21464 consensus_queue.cc:1035] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:15.796404 21381 tablet_bootstrap.cc:654] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.797592 21157 raft_consensus.cc:3037] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.800022 21329 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:15.800226 21324 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.801465 21243 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:15.806422 21112 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:15.807868 21465 raft_consensus.cc:2781] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:15.809247 21465 raft_consensus.cc:686] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:15.810243 21465 consensus_queue.cc:237] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.820761 21464 consensus_queue.cc:1035] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:15.840816 21379 tablet_bootstrap.cc:492] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:15.841362 21379 ts_tablet_manager.cc:1397] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.152s user 0.010s sys 0.014s
I20260812 19:12:15.840248 21443 consensus_queue.cc:1035] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:15.845080 21378 tablet_bootstrap.cc:654] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.848452 21379 raft_consensus.cc:348] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.850059 21379 raft_consensus.cc:374] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:15.850513 21379 raft_consensus.cc:729] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:15.855115 21379 consensus_queue.cc:260] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.863385 21379 ts_tablet_manager.cc:1428] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.022s user 0.012s sys 0.000s
I20260812 19:12:15.864470 21379 tablet_bootstrap.cc:492] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:15.862526 21390 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:15.865021 21390 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.881379 21390 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:15.886276 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:15.889153 21381 tablet_bootstrap.cc:492] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:15.889815 21381 ts_tablet_manager.cc:1397] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.141s user 0.007s sys 0.015s
I20260812 19:12:15.889264 21385 consensus_queue.cc:1035] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260812 19:12:15.896018 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:15.897447 21244 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
W20260812 19:12:15.898556 21287 leader_election.cc:341] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:15.899646 21287 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:15.901000 21390 raft_consensus.cc:2781] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:15.901575 21390 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:15.902109 21390 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:15.902721 21381 raft_consensus.cc:348] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.903628 21381 raft_consensus.cc:374] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:15.903963 21381 raft_consensus.cc:729] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:15.904613 21381 consensus_queue.cc:260] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.908694 21381 ts_tablet_manager.cc:1428] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20260812 19:12:15.914017 21443 consensus_queue.cc:1035] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:15.923352 21381 tablet_bootstrap.cc:492] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:15.942925 21416 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:15.979516 21381 tablet_bootstrap.cc:654] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:15.963883 21416 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:15.958027 21390 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.002029 21390 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:16.003430 21390 consensus_queue.cc:1035] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:16.023586 21378 tablet_bootstrap.cc:492] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:16.024339 21378 ts_tablet_manager.cc:1397] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.264s user 0.009s sys 0.035s
I20260812 19:12:16.036979 21378 raft_consensus.cc:348] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.038012 21378 raft_consensus.cc:374] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.038332 21378 raft_consensus.cc:729] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.040828 21378 consensus_queue.cc:260] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.043684 21416 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:16.050554 21381 tablet_bootstrap.cc:492] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:16.040766 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:16.023591 21036 catalog_manager.cc:5495] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:16.057592 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:16.047961 21378 ts_tablet_manager.cc:1428] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.020s user 0.008s sys 0.003s
I20260812 19:12:16.061028 21381 ts_tablet_manager.cc:1397] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.138s user 0.009s sys 0.016s
I20260812 19:12:16.053943 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:16.061585 21378 tablet_bootstrap.cc:492] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:16.062069 21379 tablet_bootstrap.cc:654] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.062914 21243 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
W20260812 19:12:16.070792 21287 leader_election.cc:341] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
W20260812 19:12:16.072213 21287 leader_election.cc:341] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:16.068871 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:16.074976 21416 consensus_queue.cc:1035] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260812 19:12:16.091466 21287 leader_election.cc:341] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:16.092727 21287 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:16.093253 21381 raft_consensus.cc:348] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.094134 21381 raft_consensus.cc:374] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.094475 21381 raft_consensus.cc:729] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.096269 21497 raft_consensus.cc:2726] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:16.116328 21378 tablet_bootstrap.cc:654] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.112820 21381 consensus_queue.cc:260] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.126178 21379 tablet_bootstrap.cc:492] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:16.126775 21379 ts_tablet_manager.cc:1397] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.262s user 0.011s sys 0.013s
I20260812 19:12:16.132129 21381 ts_tablet_manager.cc:1428] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.071s user 0.008s sys 0.003s
I20260812 19:12:16.125450 21243 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:16.144613 21379 raft_consensus.cc:348] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.145730 21379 raft_consensus.cc:374] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.146085 21379 raft_consensus.cc:729] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.151093 21379 consensus_queue.cc:260] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.156288 21379 ts_tablet_manager.cc:1428] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.029s user 0.009s sys 0.000s
I20260812 19:12:16.158011 21379 tablet_bootstrap.cc:492] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:16.165333 21381 tablet_bootstrap.cc:492] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:16.166381 21287 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:16.168604 21494 raft_consensus.cc:2781] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:16.169346 21494 raft_consensus.cc:686] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:16.175935 21494 consensus_queue.cc:237] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.195536 21379 tablet_bootstrap.cc:654] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.200834 21378 tablet_bootstrap.cc:492] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:16.202538 21378 ts_tablet_manager.cc:1397] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.141s user 0.020s sys 0.036s
I20260812 19:12:16.210801 21381 tablet_bootstrap.cc:654] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.213452 21378 raft_consensus.cc:348] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.212532 21379 tablet_bootstrap.cc:492] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:16.214389 21378 raft_consensus.cc:374] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.214816 21378 raft_consensus.cc:729] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.214975 21379 ts_tablet_manager.cc:1397] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.057s user 0.008s sys 0.013s
I20260812 19:12:16.216250 21378 consensus_queue.cc:260] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.226313 21378 ts_tablet_manager.cc:1428] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.023s user 0.005s sys 0.002s
I20260812 19:12:16.227200 21378 tablet_bootstrap.cc:492] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:16.220714 21379 raft_consensus.cc:348] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.228065 21379 raft_consensus.cc:374] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.228283 21379 raft_consensus.cc:729] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.229187 21379 consensus_queue.cc:260] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.232399 21379 ts_tablet_manager.cc:1428] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260812 19:12:16.232548 21495 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:12:16.233299 21093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260812 19:12:16.233875 21495 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.237157 21379 tablet_bootstrap.cc:492] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:16.239262 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:16.239774 21378 tablet_bootstrap.cc:654] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.246079 21495 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:16.240442 21156 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:16.249604 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:16.252846 21329 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:16.251276 21203 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:16.255029 21495 raft_consensus.cc:2781] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:16.256212 21495 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:16.256831 21495 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:16.262398 21465 consensus_queue.cc:1035] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:16.259057 21034 catalog_manager.cc:5495] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:16.277172 21462 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:16.277961 21462 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.283164 21462 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:16.286475 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:12:16.288894 21112 leader_election.cc:341] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:16.286670 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:16.290155 21244 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:16.290776 21495 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.291798 21114 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:12:16.294023 21462 raft_consensus.cc:2781] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:16.296175 21379 tablet_bootstrap.cc:654] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.296902 21378 tablet_bootstrap.cc:492] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:16.297498 21378 ts_tablet_manager.cc:1397] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.070s user 0.021s sys 0.011s
I20260812 19:12:16.294723 21462 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:16.300707 21462 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:16.302326 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:16.303531 21329 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:16.305402 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:16.307385 21156 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:16.309959 21378 raft_consensus.cc:348] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.311204 21378 raft_consensus.cc:374] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.311609 21378 raft_consensus.cc:729] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.310855 21495 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:16.312608 21378 consensus_queue.cc:260] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.315750 21381 tablet_bootstrap.cc:492] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:16.316439 21381 ts_tablet_manager.cc:1397] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.183s user 0.014s sys 0.011s
I20260812 19:12:16.317896 21378 ts_tablet_manager.cc:1428] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.020s user 0.000s sys 0.012s
I20260812 19:12:16.319096 21378 tablet_bootstrap.cc:492] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:16.315737 21329 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:16.321404 21200 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:16.323211 21495 raft_consensus.cc:2781] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:16.324368 21495 raft_consensus.cc:686] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:16.326121 21495 consensus_queue.cc:237] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.328719 21462 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.335355 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:16.336107 21378 tablet_bootstrap.cc:654] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.336460 21243 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:16.345717 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:12:16.347339 21112 leader_election.cc:341] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:16.336822 21462 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:16.349941 21381 raft_consensus.cc:348] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.350559 21381 raft_consensus.cc:374] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.350864 21381 raft_consensus.cc:729] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.354420 21381 consensus_queue.cc:260] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.360402 21156 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:16.360471 21379 tablet_bootstrap.cc:492] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:16.365087 21381 ts_tablet_manager.cc:1428] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.048s user 0.011s sys 0.000s
I20260812 19:12:16.366560 21381 tablet_bootstrap.cc:492] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:16.368733 21379 ts_tablet_manager.cc:1397] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.132s user 0.017s sys 0.029s
I20260812 19:12:16.375753 21379 raft_consensus.cc:348] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.376370 21379 raft_consensus.cc:374] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.376598 21379 raft_consensus.cc:729] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.372103 21243 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:16.374881 21419 consensus_queue.cc:1035] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260812 19:12:16.379441 21114 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:12:16.382994 21379 consensus_queue.cc:260] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.386190 21379 ts_tablet_manager.cc:1428] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260812 19:12:16.390494 21379 tablet_bootstrap.cc:492] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:16.396939 21419 raft_consensus.cc:2781] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:16.418990 21419 raft_consensus.cc:686] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:16.432860 21381 tablet_bootstrap.cc:654] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.438205 21494 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:16.439858 21494 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.430985 21419 consensus_queue.cc:237] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.445529 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:16.449542 21494 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:16.450894 21508 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:16.451357 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:12:16.453214 21287 leader_election.cc:341] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:16.453873 21379 tablet_bootstrap.cc:654] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.453214 21508 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.457824 21156 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:16.460161 21378 tablet_bootstrap.cc:492] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:16.460994 21378 ts_tablet_manager.cc:1397] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.142s user 0.018s sys 0.016s
I20260812 19:12:16.461072 21287 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:16.464638 21494 raft_consensus.cc:2781] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:16.465240 21494 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:16.465871 21494 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:16.466492 21508 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:16.467502 21508 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:16.468230 21508 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.470644 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:16.475314 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:16.491742 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:16.495837 21241 raft_consensus.cc:2379] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 1: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:12:16.499073 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:16.502000 21379 tablet_bootstrap.cc:492] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:16.502784 21379 ts_tablet_manager.cc:1397] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.113s user 0.014s sys 0.006s
I20260812 19:12:16.499853 21508 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:16.478235 21036 catalog_manager.cc:5495] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:16.524708 21378 raft_consensus.cc:348] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.525754 21378 raft_consensus.cc:374] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.526069 21378 raft_consensus.cc:729] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.526890 21378 consensus_queue.cc:260] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:16.527729 21114 leader_election.cc:341] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:16.523393 21112 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:16.531448 21379 raft_consensus.cc:348] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.532122 21419 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:16.532469 21379 raft_consensus.cc:374] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.533129 21379 raft_consensus.cc:729] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.532444 21378 ts_tablet_manager.cc:1428] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.071s user 0.010s sys 0.018s
I20260812 19:12:16.533030 21034 catalog_manager.cc:5495] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:16.537462 21378 tablet_bootstrap.cc:492] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:16.535912 21379 consensus_queue.cc:260] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.540536 21379 ts_tablet_manager.cc:1428] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.037s user 0.009s sys 0.007s
I20260812 19:12:16.545121 21379 tablet_bootstrap.cc:492] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:16.545392 21381 tablet_bootstrap.cc:492] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:16.545902 21381 ts_tablet_manager.cc:1397] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.180s user 0.021s sys 0.012s
I20260812 19:12:16.549531 21381 raft_consensus.cc:348] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.554008 21378 tablet_bootstrap.cc:654] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.559736 21419 raft_consensus.cc:2726] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:16.523089 21494 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.559089 21379 tablet_bootstrap.cc:654] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.550375 21381 raft_consensus.cc:374] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.561805 21381 raft_consensus.cc:729] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.564170 21494 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:16.570259 21327 raft_consensus.cc:2379] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 1: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:12:16.569319 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:16.571895 21156 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:16.568156 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:16.573776 21243 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:16.574851 21381 consensus_queue.cc:260] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.575377 21112 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:16.587168 21381 ts_tablet_manager.cc:1428] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.041s user 0.008s sys 0.002s
I20260812 19:12:16.589032 21381 tablet_bootstrap.cc:492] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:16.593912 21243 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:16.595872 21287 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:16.596333 21378 tablet_bootstrap.cc:492] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:16.597075 21378 ts_tablet_manager.cc:1397] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.060s user 0.022s sys 0.011s
I20260812 19:12:16.597154 21381 tablet_bootstrap.cc:654] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.597100 21494 raft_consensus.cc:2781] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:16.598273 21494 raft_consensus.cc:686] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:16.599807 21494 consensus_queue.cc:237] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.600694 21156 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:16.601552 21419 raft_consensus.cc:2726] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:16.601488 21378 raft_consensus.cc:348] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.602671 21378 raft_consensus.cc:374] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.603132 21378 raft_consensus.cc:729] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.604053 21378 consensus_queue.cc:260] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.610015 21379 tablet_bootstrap.cc:492] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:16.610569 21379 ts_tablet_manager.cc:1397] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.066s user 0.011s sys 0.005s
I20260812 19:12:16.612924 21495 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:16.613952 21497 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:16.614662 21378 ts_tablet_manager.cc:1428] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.017s user 0.005s sys 0.004s
I20260812 19:12:16.614467 21497 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.615865 21378 tablet_bootstrap.cc:492] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:16.613962 21495 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.619841 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:16.620766 21497 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:16.627439 21379 raft_consensus.cc:348] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.628342 21379 raft_consensus.cc:374] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.628647 21379 raft_consensus.cc:729] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.632346 21379 consensus_queue.cc:260] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.634811 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:16.637504 21379 ts_tablet_manager.cc:1428] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.027s user 0.007s sys 0.001s
I20260812 19:12:16.638773 21379 tablet_bootstrap.cc:492] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:16.638947 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:16.639786 21327 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:16.636466 21243 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 1: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:12:16.642375 21200 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:16.644789 21495 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:16.646570 21443 raft_consensus.cc:2781] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:16.649266 21443 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:16.649796 21443 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:16.647876 21287 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:16.653441 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:16.655205 21203 leader_election.cc:341] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:16.658522 21379 tablet_bootstrap.cc:654] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.658475 21497 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:16.665773 21378 tablet_bootstrap.cc:654] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.665648 21385 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:16.665447 21034 catalog_manager.cc:5495] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:16.666323 21385 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.669399 21443 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.671175 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:16.672605 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:16.673069 21200 leader_election.cc:341] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:16.671993 21385 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:12:16.674678 21203 leader_election.cc:341] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:16.675326 21203 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:16.676155 21443 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:16.680686 21381 tablet_bootstrap.cc:492] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:16.681257 21381 ts_tablet_manager.cc:1397] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.092s user 0.014s sys 0.022s
I20260812 19:12:16.677842 21385 raft_consensus.cc:2726] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:16.684206 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:16.684942 21497 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:16.686297 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:12:16.685896 21203 leader_election.cc:341] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:16.686949 21327 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:16.685448 21381 raft_consensus.cc:348] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.691859 21381 raft_consensus.cc:374] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.692257 21381 raft_consensus.cc:729] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.693008 21381 consensus_queue.cc:260] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.695081 21381 ts_tablet_manager.cc:1428] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.014s user 0.006s sys 0.002s
I20260812 19:12:16.696106 21497 consensus_queue.cc:1035] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:16.696534 21327 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:16.697875 21200 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:16.704619 21385 raft_consensus.cc:2781] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:16.705061 21381 tablet_bootstrap.cc:492] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:16.705572 21385 raft_consensus.cc:686] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:16.707296 21385 consensus_queue.cc:237] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.715663 21379 tablet_bootstrap.cc:492] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:16.716439 21379 ts_tablet_manager.cc:1397] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.078s user 0.011s sys 0.011s
I20260812 19:12:16.721342 21379 raft_consensus.cc:348] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.722610 21379 raft_consensus.cc:374] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.722798 21381 tablet_bootstrap.cc:654] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.722993 21379 raft_consensus.cc:729] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.723874 21379 consensus_queue.cc:260] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.724038 21497 consensus_queue.cc:1035] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260812 19:12:16.728422 21379 ts_tablet_manager.cc:1428] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20260812 19:12:16.728981 21378 tablet_bootstrap.cc:492] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:16.729426 21379 tablet_bootstrap.cc:492] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:16.729465 21378 ts_tablet_manager.cc:1397] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.114s user 0.011s sys 0.023s
I20260812 19:12:16.744596 21378 raft_consensus.cc:348] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.745653 21378 raft_consensus.cc:374] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.745936 21378 raft_consensus.cc:729] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.746135 21443 consensus_queue.cc:1035] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:16.746908 21378 consensus_queue.cc:260] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.749270 21378 ts_tablet_manager.cc:1428] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260812 19:12:16.750487 21378 tablet_bootstrap.cc:492] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:16.757793 21379 tablet_bootstrap.cc:654] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.797086 21035 catalog_manager.cc:5495] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:16.818508 21378 tablet_bootstrap.cc:654] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.836019 21381 tablet_bootstrap.cc:492] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:16.836580 21381 ts_tablet_manager.cc:1397] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.132s user 0.017s sys 0.018s
I20260812 19:12:16.809821 21494 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:16.842156 21381 raft_consensus.cc:348] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.842980 21381 raft_consensus.cc:374] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.843264 21381 raft_consensus.cc:729] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.844024 21381 consensus_queue.cc:260] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.838122 21533 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:16.852737 21495 consensus_queue.cc:1035] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:16.853361 21533 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.848675 21381 ts_tablet_manager.cc:1428] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.012s user 0.004s sys 0.004s
I20260812 19:12:16.856993 21381 tablet_bootstrap.cc:492] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:16.856513 21494 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.866676 21533 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:16.868764 21494 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:16.869274 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:16.872133 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:16.876799 21287 leader_election.cc:341] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260812 19:12:16.877717 21287 leader_election.cc:341] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:16.878194 21287 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:16.879623 21494 raft_consensus.cc:2726] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:16.879715 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:16.880555 21241 raft_consensus.cc:2454] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:16.881906 21287 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:16.882889 21494 raft_consensus.cc:2781] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:16.883363 21494 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:16.883797 21494 raft_consensus.cc:3037] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:16.888540 21508 consensus_queue.cc:1035] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260812 19:12:16.888777 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:16.912101 21287 leader_election.cc:341] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:16.915071 21379 tablet_bootstrap.cc:492] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:16.915584 21379 ts_tablet_manager.cc:1397] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.186s user 0.027s sys 0.021s
I20260812 19:12:16.920881 21379 raft_consensus.cc:348] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.921751 21379 raft_consensus.cc:374] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.922088 21379 raft_consensus.cc:729] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.922888 21379 consensus_queue.cc:260] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.924971 21378 tablet_bootstrap.cc:492] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:16.925488 21378 ts_tablet_manager.cc:1397] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.175s user 0.021s sys 0.026s
I20260812 19:12:16.925555 21379 ts_tablet_manager.cc:1428] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20260812 19:12:16.926777 21379 tablet_bootstrap.cc:492] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:16.928133 21494 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.933482 21378 raft_consensus.cc:348] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.934520 21378 raft_consensus.cc:374] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.935261 21508 consensus_queue.cc:1035] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:16.935917 21378 raft_consensus.cc:729] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.936780 21378 consensus_queue.cc:260] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.939041 21379 tablet_bootstrap.cc:654] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.937263 21494 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:16.940178 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:16.942080 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:16.943295 21241 raft_consensus.cc:3037] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:16.964390 21381 tablet_bootstrap.cc:654] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.971626 21378 ts_tablet_manager.cc:1428] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.041s user 0.009s sys 0.000s
I20260812 19:12:16.974979 21378 tablet_bootstrap.cc:492] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:16.983105 21241 raft_consensus.cc:2454] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
W20260812 19:12:16.978119 21287 leader_election.cc:341] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:16.985001 21287 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:16.989137 21379 tablet_bootstrap.cc:492] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:16.989042 21494 raft_consensus.cc:2781] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:16.989711 21379 ts_tablet_manager.cc:1397] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.063s user 0.013s sys 0.014s
I20260812 19:12:16.989711 21494 raft_consensus.cc:686] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:16.991088 21494 consensus_queue.cc:237] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.996527 21378 tablet_bootstrap.cc:654] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:16.997519 21379 raft_consensus.cc:348] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:16.998605 21379 raft_consensus.cc:374] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:16.998994 21379 raft_consensus.cc:729] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:16.999879 21379 consensus_queue.cc:260] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.004907 21497 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:17.006240 21497 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.008400 21379 ts_tablet_manager.cc:1428] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260812 19:12:17.010325 21497 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:17.010866 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:17.014258 21287 leader_election.cc:341] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.018841 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:12:17.021240 21287 leader_election.cc:341] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.022116 21287 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:17.023406 21497 raft_consensus.cc:2726] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:17.027915 21512 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:17.028509 21512 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.030952 21379 tablet_bootstrap.cc:492] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:17.042393 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:17.045233 21512 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:17.047494 21381 tablet_bootstrap.cc:492] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:17.047108 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:17.048133 21381 ts_tablet_manager.cc:1397] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.191s user 0.006s sys 0.028s
I20260812 19:12:17.048508 21324 raft_consensus.cc:2379] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 1: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:12:17.049630 21379 tablet_bootstrap.cc:654] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.054941 21381 raft_consensus.cc:348] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.055859 21381 raft_consensus.cc:374] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.051043 21112 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:17.056123 21381 raft_consensus.cc:729] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.060691 21378 tablet_bootstrap.cc:492] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:17.056607 21512 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.064589 21035 catalog_manager.cc:5495] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:17.066272 21378 ts_tablet_manager.cc:1397] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.091s user 0.017s sys 0.018s
I20260812 19:12:17.070163 21378 raft_consensus.cc:348] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.071574 21378 raft_consensus.cc:374] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.071967 21378 raft_consensus.cc:729] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.073035 21419 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:17.072865 21378 consensus_queue.cc:260] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.073756 21419 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.078583 21378 ts_tablet_manager.cc:1428] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.012s user 0.006s sys 0.002s
I20260812 19:12:17.079221 21419 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:17.080000 21378 tablet_bootstrap.cc:492] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:17.082923 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:12:17.085368 21114 leader_election.cc:341] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.083132 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:12:17.088325 21112 leader_election.cc:341] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.089131 21112 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:17.090842 21378 tablet_bootstrap.cc:654] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.093624 21508 raft_consensus.cc:2726] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:17.094894 21381 consensus_queue.cc:260] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.106808 21378 tablet_bootstrap.cc:492] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:17.107496 21378 ts_tablet_manager.cc:1397] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.028s user 0.024s sys 0.002s
I20260812 19:12:17.111721 21378 raft_consensus.cc:348] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.112900 21378 raft_consensus.cc:374] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.113263 21378 raft_consensus.cc:729] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.114138 21378 consensus_queue.cc:260] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.117605 21381 ts_tablet_manager.cc:1428] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.069s user 0.006s sys 0.003s
I20260812 19:12:17.118747 21381 tablet_bootstrap.cc:492] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:17.126369 21378 ts_tablet_manager.cc:1428] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.018s user 0.009s sys 0.000s
I20260812 19:12:17.131558 21378 tablet_bootstrap.cc:492] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:17.136445 21494 consensus_queue.cc:1035] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:17.140511 21381 tablet_bootstrap.cc:654] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.142274 21379 tablet_bootstrap.cc:492] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:17.143090 21379 ts_tablet_manager.cc:1397] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.112s user 0.040s sys 0.042s
I20260812 19:12:17.155723 21512 raft_consensus.cc:2726] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:17.148909 21379 raft_consensus.cc:348] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.157203 21379 raft_consensus.cc:374] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.157563 21379 raft_consensus.cc:729] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.158334 21379 consensus_queue.cc:260] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.161489 21379 ts_tablet_manager.cc:1428] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20260812 19:12:17.162647 21379 tablet_bootstrap.cc:492] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:17.166271 21495 consensus_queue.cc:1035] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:17.167403 21385 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:17.168110 21385 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.171056 21385 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:17.172775 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:12:17.174644 21200 leader_election.cc:341] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.176239 21378 tablet_bootstrap.cc:654] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.176239 21379 tablet_bootstrap.cc:654] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.177325 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:17.182796 21203 leader_election.cc:341] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.183698 21203 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:17.185789 21385 raft_consensus.cc:2726] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:17.189891 21381 tablet_bootstrap.cc:492] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:17.190644 21381 ts_tablet_manager.cc:1397] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.072s user 0.010s sys 0.004s
I20260812 19:12:17.200739 21497 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:17.201355 21497 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.202147 21529 consensus_queue.cc:1035] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260812 19:12:17.208072 21497 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:17.208992 21381 raft_consensus.cc:348] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.209730 21381 raft_consensus.cc:374] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.209969 21381 raft_consensus.cc:729] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.210654 21381 consensus_queue.cc:260] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.211892 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:17.214923 21381 ts_tablet_manager.cc:1428] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.024s user 0.006s sys 0.002s
I20260812 19:12:17.215705 21381 tablet_bootstrap.cc:492] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
W20260812 19:12:17.218680 21287 leader_election.cc:341] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.219583 21378 tablet_bootstrap.cc:492] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:17.220120 21378 ts_tablet_manager.cc:1397] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.089s user 0.016s sys 0.013s
I20260812 19:12:17.221772 21443 consensus_queue.cc:1035] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:12:17.223544 21378 raft_consensus.cc:348] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.251652 21378 raft_consensus.cc:374] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.252137 21378 raft_consensus.cc:729] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.226256 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:17.259294 21378 consensus_queue.cc:260] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.262336 21378 ts_tablet_manager.cc:1428] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.042s user 0.006s sys 0.002s
I20260812 19:12:17.263423 21378 tablet_bootstrap.cc:492] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
W20260812 19:12:17.268139 21287 leader_election.cc:341] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.269181 21287 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:17.274761 21379 tablet_bootstrap.cc:492] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:17.275403 21379 ts_tablet_manager.cc:1397] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.113s user 0.024s sys 0.009s
I20260812 19:12:17.275959 21497 raft_consensus.cc:2726] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:17.279189 21379 raft_consensus.cc:348] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.279904 21381 tablet_bootstrap.cc:654] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.280186 21379 raft_consensus.cc:374] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.280575 21379 raft_consensus.cc:729] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.281684 21379 consensus_queue.cc:260] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.284340 21379 ts_tablet_manager.cc:1428] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260812 19:12:17.285555 21379 tablet_bootstrap.cc:492] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:17.298199 21512 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:17.298921 21512 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.306985 21512 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:17.308019 21378 tablet_bootstrap.cc:654] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.307621 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:17.308689 21241 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:17.310108 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:17.315168 21381 tablet_bootstrap.cc:492] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:17.315729 21381 ts_tablet_manager.cc:1397] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.100s user 0.010s sys 0.014s
I20260812 19:12:17.315312 21114 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:17.316846 21512 raft_consensus.cc:2781] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260812 19:12:17.316923 21112 leader_election.cc:341] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.317452 21512 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:17.318208 21512 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.330266 21379 tablet_bootstrap.cc:654] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.334236 21419 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:17.336177 21419 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.340970 21419 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:17.343509 21381 raft_consensus.cc:348] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.346647 21381 raft_consensus.cc:374] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.344534 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:17.347929 21324 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:17.346939 21381 raft_consensus.cc:729] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.350188 21379 tablet_bootstrap.cc:492] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:17.350853 21379 ts_tablet_manager.cc:1397] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.066s user 0.017s sys 0.000s
I20260812 19:12:17.349496 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:17.354184 21244 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
W20260812 19:12:17.354358 21093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260812 19:12:17.353302 21112 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:17.357090 21419 raft_consensus.cc:2781] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:17.357573 21419 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:17.357955 21419 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.350896 21381 consensus_queue.cc:260] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.355105 21379 raft_consensus.cc:348] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.355811 21512 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.358141 21378 tablet_bootstrap.cc:492] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:17.360543 21378 ts_tablet_manager.cc:1397] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.097s user 0.005s sys 0.033s
I20260812 19:12:17.362542 21381 ts_tablet_manager.cc:1428] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.047s user 0.010s sys 0.000s
I20260812 19:12:17.364207 21512 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:17.364132 21378 raft_consensus.cc:348] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.364789 21379 raft_consensus.cc:374] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.365372 21378 raft_consensus.cc:374] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.365754 21378 raft_consensus.cc:729] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.366145 21381 tablet_bootstrap.cc:492] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:17.366768 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:17.366518 21378 consensus_queue.cc:260] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:17.368597 21112 leader_election.cc:341] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.365500 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:17.370298 21244 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.372079 21419 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.375353 21378 ts_tablet_manager.cc:1428] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260812 19:12:17.375970 21381 tablet_bootstrap.cc:654] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.376426 21378 tablet_bootstrap.cc:492] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:17.384353 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:17.385251 21241 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.391858 21378 tablet_bootstrap.cc:654] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.398375 21244 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:17.365757 21379 raft_consensus.cc:729] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.400741 21114 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:12:17.400874 21379 consensus_queue.cc:260] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.404618 21379 ts_tablet_manager.cc:1428] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.053s user 0.008s sys 0.000s
I20260812 19:12:17.405691 21378 tablet_bootstrap.cc:492] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:17.406188 21378 ts_tablet_manager.cc:1397] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.010s
I20260812 19:12:17.407521 21379 tablet_bootstrap.cc:492] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:17.407457 21512 raft_consensus.cc:2781] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:17.408699 21156 raft_consensus.cc:3037] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.409727 21385 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:17.412312 21497 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:17.410246 21385 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.412938 21497 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.408660 21512 raft_consensus.cc:686] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:17.415992 21512 consensus_queue.cc:237] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.418759 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:17.419837 21324 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:17.423424 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:17.426808 21200 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:17.428596 21385 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:12:17.430444 21287 leader_election.cc:341] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.435004 21497 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:17.438704 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:17.439250 21508 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:17.440348 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:17.440064 21508 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:17.441530 21203 leader_election.cc:341] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.443399 21529 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:17.443926 21529 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.444823 21379 tablet_bootstrap.cc:654] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.447461 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:17.448303 21243 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:17.448653 21529 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:17.452883 21508 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:17.454308 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:17.454293 21495 raft_consensus.cc:2781] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:17.455893 21495 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:12:17.456301 21112 leader_election.cc:341] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.456735 21495 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.462044 21378 raft_consensus.cc:348] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.463037 21378 raft_consensus.cc:374] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.462886 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:17.463449 21378 raft_consensus.cc:729] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.464138 21378 consensus_queue.cc:260] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.466596 21378 ts_tablet_manager.cc:1428] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.060s user 0.003s sys 0.003s
W20260812 19:12:17.466703 21287 leader_election.cc:341] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.439303 21497 consensus_queue.cc:1035] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:17.469574 21378 tablet_bootstrap.cc:492] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
W20260812 19:12:17.474997 21287 leader_election.cc:341] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.475871 21287 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:17.477923 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:17.481390 21244 raft_consensus.cc:2454] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:17.488399 21497 raft_consensus.cc:2726] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:17.494398 21114 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:12:17.495971 21564 raft_consensus.cc:2781] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:17.496906 21378 tablet_bootstrap.cc:654] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.497020 21564 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:17.497476 21564 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.488018 21419 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:17.500486 21287 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:17.489359 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:17.502053 21324 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.502975 21497 raft_consensus.cc:2781] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:17.503515 21497 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:17.504006 21497 raft_consensus.cc:3037] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.519263 21381 tablet_bootstrap.cc:492] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:17.519977 21381 ts_tablet_manager.cc:1397] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.156s user 0.043s sys 0.046s
I20260812 19:12:17.525005 21379 tablet_bootstrap.cc:492] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:17.525583 21379 ts_tablet_manager.cc:1397] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.118s user 0.008s sys 0.016s
I20260812 19:12:17.531985 21379 raft_consensus.cc:348] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.532665 21379 raft_consensus.cc:374] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.532919 21379 raft_consensus.cc:729] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.533529 21379 consensus_queue.cc:260] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.535913 21381 raft_consensus.cc:348] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.536705 21381 raft_consensus.cc:374] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.537051 21381 raft_consensus.cc:729] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.543792 21381 consensus_queue.cc:260] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.539242 21564 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.545012 21324 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:17.546850 21112 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:17.548012 21497 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.549561 21512 raft_consensus.cc:2781] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:17.550154 21512 raft_consensus.cc:686] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:17.551983 21495 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.552310 21512 consensus_queue.cc:237] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.552384 21035 catalog_manager.cc:5495] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:17.555181 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:17.555886 21157 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.551258 21381 ts_tablet_manager.cc:1428] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.029s user 0.004s sys 0.003s
I20260812 19:12:17.560621 21381 tablet_bootstrap.cc:492] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:17.550669 21379 ts_tablet_manager.cc:1428] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.025s user 0.007s sys 0.005s
I20260812 19:12:17.564555 21379 tablet_bootstrap.cc:492] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:17.572258 21497 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:17.573295 21495 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:17.550475 21241 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:17.576558 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:17.577242 21324 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.577116 21497 consensus_queue.cc:1035] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:17.579011 21381 tablet_bootstrap.cc:654] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.579865 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:17.581498 21244 raft_consensus.cc:3037] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.582913 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:17.591693 21379 tablet_bootstrap.cc:654] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.593806 21378 tablet_bootstrap.cc:492] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:17.594290 21378 ts_tablet_manager.cc:1397] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.125s user 0.015s sys 0.015s
I20260812 19:12:17.591790 21324 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:17.598762 21157 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:17.598929 21378 raft_consensus.cc:348] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.599328 21200 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:17.599758 21378 raft_consensus.cc:374] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.600033 21378 raft_consensus.cc:729] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.600653 21443 raft_consensus.cc:2781] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:17.600795 21378 consensus_queue.cc:260] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.601193 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:17.601909 21241 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.601233 21443 raft_consensus.cc:686] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:17.604133 21443 consensus_queue.cc:237] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.608428 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:17.609311 21327 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.611374 21378 ts_tablet_manager.cc:1428] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
I20260812 19:12:17.612597 21378 tablet_bootstrap.cc:492] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:17.615521 21497 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:17.616207 21497 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.618489 21497 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:17.618754 21564 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:17.619234 21378 tablet_bootstrap.cc:654] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.620347 21244 raft_consensus.cc:2454] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:17.622249 21327 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
W20260812 19:12:17.622705 21287 leader_election.cc:341] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.624064 21112 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:17.630230 21379 tablet_bootstrap.cc:492] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:17.631488 21419 raft_consensus.cc:2781] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:17.634038 21379 ts_tablet_manager.cc:1397] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.070s user 0.006s sys 0.008s
I20260812 19:12:17.654942 21287 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:17.634877 21419 raft_consensus.cc:686] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:17.665202 21381 tablet_bootstrap.cc:492] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:17.633383 21241 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:17.667989 21381 ts_tablet_manager.cc:1397] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.108s user 0.016s sys 0.000s
I20260812 19:12:17.668047 21419 consensus_queue.cc:237] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.648234 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:17.649693 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:17.665930 21379 raft_consensus.cc:348] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.671500 21379 raft_consensus.cc:374] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.671314 21497 raft_consensus.cc:2781] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:17.671355 21244 raft_consensus.cc:2454] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:17.672796 21381 raft_consensus.cc:348] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.672072 21497 raft_consensus.cc:686] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:17.674521 21497 consensus_queue.cc:237] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.671865 21379 raft_consensus.cc:729] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
W20260812 19:12:17.672191 21287 leader_election.cc:341] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.677386 21379 consensus_queue.cc:260] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.679086 21287 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:17.679440 21378 tablet_bootstrap.cc:492] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:17.673559 21381 raft_consensus.cc:374] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.679898 21381 raft_consensus.cc:729] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.680094 21378 ts_tablet_manager.cc:1397] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.068s user 0.012s sys 0.005s
I20260812 19:12:17.680627 21381 consensus_queue.cc:260] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.681347 21379 ts_tablet_manager.cc:1428] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.018s user 0.003s sys 0.004s
I20260812 19:12:17.682058 21529 raft_consensus.cc:2781] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:17.682519 21529 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:17.682922 21529 raft_consensus.cc:3037] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.683178 21379 tablet_bootstrap.cc:492] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:17.683768 21378 raft_consensus.cc:348] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.684916 21378 raft_consensus.cc:374] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.685256 21378 raft_consensus.cc:729] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.686345 21378 consensus_queue.cc:260] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.665884 21036 catalog_manager.cc:5495] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:17.689334 21378 ts_tablet_manager.cc:1428] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260812 19:12:17.690766 21495 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:17.691412 21495 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.694079 21381 ts_tablet_manager.cc:1428] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.026s user 0.004s sys 0.003s
I20260812 19:12:17.695164 21381 tablet_bootstrap.cc:492] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:17.696789 21495 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:17.697691 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:17.698263 21378 tablet_bootstrap.cc:492] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:17.698483 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:17.699693 21327 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:17.698527 21157 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:17.705433 21200 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:17.710011 21495 raft_consensus.cc:2781] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:17.711529 21495 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:17.712154 21378 tablet_bootstrap.cc:654] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.712054 21495 raft_consensus.cc:3037] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.726648 21381 tablet_bootstrap.cc:654] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.731690 21035 catalog_manager.cc:5495] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:17.735363 21529 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.738868 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:17.739616 21157 raft_consensus.cc:3037] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.745440 21495 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.749001 21379 tablet_bootstrap.cc:654] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.749066 21157 raft_consensus.cc:2454] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:17.752755 21495 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:17.755501 21378 tablet_bootstrap.cc:492] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:17.764724 21378 ts_tablet_manager.cc:1397] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.074s user 0.019s sys 0.032s
I20260812 19:12:17.755116 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:17.754714 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:17.754428 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:17.755575 21287 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:17.767151 21154 raft_consensus.cc:3037] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.768021 21497 raft_consensus.cc:2781] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:17.767700 21327 raft_consensus.cc:3037] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.768596 21497 raft_consensus.cc:686] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:17.753415 21529 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:17.767151 21241 raft_consensus.cc:3037] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.774643 21497 consensus_queue.cc:237] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.788309 21378 raft_consensus.cc:348] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.789108 21378 raft_consensus.cc:374] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.789425 21378 raft_consensus.cc:729] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.790103 21378 consensus_queue.cc:260] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.787603 21034 catalog_manager.cc:5495] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:17.808624 21381 tablet_bootstrap.cc:492] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:17.809161 21381 ts_tablet_manager.cc:1397] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.114s user 0.014s sys 0.005s
I20260812 19:12:17.801069 21036 catalog_manager.cc:5495] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:17.820134 21241 raft_consensus.cc:2454] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:17.821436 21381 raft_consensus.cc:348] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.822337 21381 raft_consensus.cc:374] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.823233 21381 raft_consensus.cc:729] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.817574 21154 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:17.828367 21381 consensus_queue.cc:260] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.832001 21203 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:17.832348 21381 ts_tablet_manager.cc:1428] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.023s user 0.008s sys 0.002s
I20260812 19:12:17.835119 21443 raft_consensus.cc:2781] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:17.836050 21381 tablet_bootstrap.cc:492] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:17.836508 21378 ts_tablet_manager.cc:1428] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.071s user 0.007s sys 0.002s
I20260812 19:12:17.835667 21443 raft_consensus.cc:686] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:17.835803 21327 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:17.837942 21443 consensus_queue.cc:237] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.849529 21379 tablet_bootstrap.cc:492] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:17.850104 21379 ts_tablet_manager.cc:1397] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.168s user 0.031s sys 0.020s
I20260812 19:12:17.852507 21381 tablet_bootstrap.cc:654] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.852535 21379 raft_consensus.cc:348] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.853256 21378 tablet_bootstrap.cc:492] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:17.853511 21379 raft_consensus.cc:374] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.853847 21379 raft_consensus.cc:729] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.854590 21379 consensus_queue.cc:260] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.855875 21035 catalog_manager.cc:5495] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:17.858598 21379 ts_tablet_manager.cc:1428] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.008s user 0.000s sys 0.008s
I20260812 19:12:17.859678 21379 tablet_bootstrap.cc:492] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:17.867556 21379 tablet_bootstrap.cc:654] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.874787 21379 tablet_bootstrap.cc:492] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:17.875375 21379 ts_tablet_manager.cc:1397] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20260812 19:12:17.878648 21379 raft_consensus.cc:348] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.879509 21379 raft_consensus.cc:374] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.879819 21379 raft_consensus.cc:729] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.880635 21379 consensus_queue.cc:260] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.883765 21379 ts_tablet_manager.cc:1428] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260812 19:12:17.883765 21378 tablet_bootstrap.cc:654] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.885927 21379 tablet_bootstrap.cc:492] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:17.886049 21035 catalog_manager.cc:5495] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:17.891309 21381 tablet_bootstrap.cc:492] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:17.891971 21381 ts_tablet_manager.cc:1397] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.057s user 0.019s sys 0.014s
I20260812 19:12:17.894513 21379 tablet_bootstrap.cc:654] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.895066 21381 raft_consensus.cc:348] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.895958 21381 raft_consensus.cc:374] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.896310 21381 raft_consensus.cc:729] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.897091 21381 consensus_queue.cc:260] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.908250 21381 ts_tablet_manager.cc:1428] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20260812 19:12:17.909472 21381 tablet_bootstrap.cc:492] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:17.916055 21381 tablet_bootstrap.cc:654] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.919404 21327 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.920464 21443 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:17.921217 21443 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.925586 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:17.929666 21381 tablet_bootstrap.cc:492] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:17.930227 21381 ts_tablet_manager.cc:1397] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.005s
W20260812 19:12:17.929721 21203 leader_election.cc:341] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.933488 21443 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:17.935313 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:17.933142 21381 raft_consensus.cc:348] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.936062 21381 raft_consensus.cc:374] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.936488 21381 raft_consensus.cc:729] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.936103 21324 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:17.937235 21381 consensus_queue.cc:260] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.938903 21200 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:17.940907 21381 ts_tablet_manager.cc:1428] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20260812 19:12:17.941960 21381 tablet_bootstrap.cc:492] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:17.942771 21443 raft_consensus.cc:2781] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:17.940842 21529 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:17.943384 21443 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:17.943845 21443 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.943797 21529 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.942555 21419 consensus_queue.cc:1035] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:17.947741 21378 tablet_bootstrap.cc:492] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:17.948266 21378 ts_tablet_manager.cc:1397] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.095s user 0.020s sys 0.017s
I20260812 19:12:17.948997 21419 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:17.949304 21385 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:17.949518 21419 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.949896 21385 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.951270 21378 raft_consensus.cc:348] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.952008 21378 raft_consensus.cc:374] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.952292 21378 raft_consensus.cc:729] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.952366 21385 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:17.953761 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:17.954510 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:17.955750 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:17.956617 21241 raft_consensus.cc:2379] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 1: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:12:17.954556 21157 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:17.960311 21379 tablet_bootstrap.cc:492] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:17.960938 21379 ts_tablet_manager.cc:1397] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.075s user 0.031s sys 0.011s
I20260812 19:12:17.960990 21443 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.963359 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:17.963749 21443 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:12:17.956377 21200 leader_election.cc:341] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:17.964582 21379 raft_consensus.cc:348] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.965231 21112 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:17.965546 21379 raft_consensus.cc:374] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:17.965942 21379 raft_consensus.cc:729] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:17.965349 21200 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:12:17.966286 21512 raft_consensus.cc:3037] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.967175 21379 consensus_queue.cc:260] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.968417 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:17.969080 21324 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.969399 21379 ts_tablet_manager.cc:1428] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260812 19:12:17.970435 21379 tablet_bootstrap.cc:492] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:17.974154 21443 raft_consensus.cc:2781] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:17.974757 21443 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:17.972965 21419 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:17.975370 21443 raft_consensus.cc:3037] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:17.977706 21379 tablet_bootstrap.cc:654] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.952988 21378 consensus_queue.cc:260] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.978271 21381 tablet_bootstrap.cc:654] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:17.975354 21512 raft_consensus.cc:2726] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:17.985694 21378 ts_tablet_manager.cc:1428] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.037s user 0.004s sys 0.003s
I20260812 19:12:17.987062 21378 tablet_bootstrap.cc:492] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:17.988723 21443 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:17.992038 21443 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:18.006913 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:18.069389 21379 tablet_bootstrap.cc:492] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:18.070092 21379 ts_tablet_manager.cc:1397] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.100s user 0.013s sys 0.002s
I20260812 19:12:18.035400 21378 tablet_bootstrap.cc:654] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:18.013916 21529 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:18.024506 21497 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:18.073055 21508 consensus_queue.cc:1035] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:18.070581 21419 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:18.074157 21497 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.074249 21379 raft_consensus.cc:348] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.075855 21379 raft_consensus.cc:374] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:18.076169 21379 raft_consensus.cc:729] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:18.008780 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:18.076838 21379 consensus_queue.cc:260] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.075029 21419 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.058251 21443 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:18.083809 21443 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.085129 21419 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:18.085795 21419 consensus_queue.cc:1035] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:12:18.086123 21443 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:18.008231 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:18.087441 21379 ts_tablet_manager.cc:1428] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260812 19:12:18.090049 21379 tablet_bootstrap.cc:492] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
W20260812 19:12:18.008231 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 49013.3us
I20260812 19:12:18.031466 21324 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:18.037760 21512 consensus_queue.cc:1035] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.050s
I20260812 19:12:18.118861 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:18.120676 21320 raft_consensus.cc:2454] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:18.128883 21497 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:18.138409 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:18.148175 21381 tablet_bootstrap.cc:492] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:18.148671 21381 ts_tablet_manager.cc:1397] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.207s user 0.020s sys 0.015s
I20260812 19:12:18.131664 21241 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:18.142256 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:18.154908 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:18.145221 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:18.156051 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:18.149096 21497 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:18.160806 21200 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:18.160667 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:18.139348 21318 raft_consensus.cc:2454] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:18.161991 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:18.160087 21244 raft_consensus.cc:2454] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:18.164979 21154 raft_consensus.cc:3037] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:18.171404 21158 raft_consensus.cc:2454] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:18.171847 21497 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:18.180037 21287 leader_election.cc:341] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:18.180230 21114 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:18.184612 21381 raft_consensus.cc:348] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.185770 21381 raft_consensus.cc:374] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:18.186131 21381 raft_consensus.cc:729] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:18.187126 21381 consensus_queue.cc:260] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.189147 21570 raft_consensus.cc:2781] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:18.189666 21564 raft_consensus.cc:2781] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:18.192521 21564 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:18.193014 21564 raft_consensus.cc:3037] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:18.203676 21570 raft_consensus.cc:686] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:18.204694 21570 consensus_queue.cc:237] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.245986 21381 ts_tablet_manager.cc:1428] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.097s user 0.009s sys 0.000s
W20260812 19:12:18.247867 21203 leader_election.cc:341] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
W20260812 19:12:18.202098 21200 leader_election.cc:341] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
W20260812 19:12:18.248653 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 49418.1us
I20260812 19:12:18.248958 21379 tablet_bootstrap.cc:654] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:18.249051 21203 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:18.252198 21385 raft_consensus.cc:2781] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:18.252697 21385 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:18.253172 21385 raft_consensus.cc:3037] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:18.262151 21378 tablet_bootstrap.cc:492] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:18.262894 21378 ts_tablet_manager.cc:1397] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.276s user 0.026s sys 0.015s
I20260812 19:12:18.277364 21378 raft_consensus.cc:348] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.278502 21378 raft_consensus.cc:374] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:18.285936 21378 raft_consensus.cc:729] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:18.286947 21378 consensus_queue.cc:260] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.307953 21378 ts_tablet_manager.cc:1428] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.045s user 0.011s sys 0.000s
I20260812 19:12:18.309336 21378 tablet_bootstrap.cc:492] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:18.312757 21381 tablet_bootstrap.cc:492] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:18.326285 21379 tablet_bootstrap.cc:492] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:18.326844 21379 ts_tablet_manager.cc:1397] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.237s user 0.006s sys 0.028s
W20260812 19:12:18.304045 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 53157.5us
I20260812 19:12:18.342880 21494 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:18.328881 21497 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:12:18.314239 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 64548.4us
I20260812 19:12:18.343848 21494 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.336311 21154 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:18.397109 21443 consensus_queue.cc:1035] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260812 19:12:18.399194 21203 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:12:18.424578 21598 consensus_queue.cc:1035] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260812 19:12:18.422541 21595 consensus_queue.cc:1035] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260812 19:12:18.435906 21443 raft_consensus.cc:2781] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:18.442119 21508 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:18.466992 21379 raft_consensus.cc:348] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.469662 21379 raft_consensus.cc:374] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:18.470104 21379 raft_consensus.cc:729] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:18.471668 21379 consensus_queue.cc:260] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:18.327978 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 51739.8us
I20260812 19:12:18.457372 21443 raft_consensus.cc:686] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:18.494654 21564 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.496052 21385 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.514482 21564 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:18.473304 21508 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.489928 21494 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:18.522385 21385 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:18.534790 21381 tablet_bootstrap.cc:654] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:18.483572 21570 consensus_queue.cc:1035] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20260812 19:12:18.549188 21378 tablet_bootstrap.cc:654] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:18.556565 21508 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:12:18.547595 21605 log.cc:924] Time spent T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260812 19:12:18.566390 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:18.567453 21241 raft_consensus.cc:2379] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 1: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:12:18.572489 21379 ts_tablet_manager.cc:1428] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.245s user 0.057s sys 0.085s
I20260812 19:12:18.548097 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:18.573820 21379 tablet_bootstrap.cc:492] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:18.574172 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:18.538692 21443 consensus_queue.cc:237] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.611027 21612 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:18.611794 21612 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.617085 21036 catalog_manager.cc:5495] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:18.618170 21612 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:18.576841 21287 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:18.622867 21494 raft_consensus.cc:2781] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:12:18.623375 21494 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:12:18.623101 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 58473.7us
I20260812 19:12:18.623839 21494 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
W20260812 19:12:18.624266 21287 leader_election.cc:341] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:18.624684 21287 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:18.626441 21529 raft_consensus.cc:2726] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:18.578692 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:18.649704 21495 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:18.613245 21155 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:18.673561 21495 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.713995 21601 consensus_queue.cc:1035] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.072s
I20260812 19:12:18.653055 21237 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:18.730705 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:12:18.743359 21287 leader_election.cc:341] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:18.752601 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:18.756809 21323 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:18.746519 21529 consensus_queue.cc:1035] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260812 19:12:18.765297 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:18.766345 21320 raft_consensus.cc:3037] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:18.767030 21378 tablet_bootstrap.cc:492] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:18.767525 21378 ts_tablet_manager.cc:1397] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.458s user 0.046s sys 0.059s
I20260812 19:12:18.772562 21495 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:18.774798 21378 raft_consensus.cc:348] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.776173 21378 raft_consensus.cc:374] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:18.780031 21378 raft_consensus.cc:729] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:18.781296 21378 consensus_queue.cc:260] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.777432 21494 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:18.831174 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 88281.1us
I20260812 19:12:18.839861 21114 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:18.842540 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:18.840709 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:18.844870 21158 raft_consensus.cc:2379] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 1: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:12:18.850314 21287 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:18.845811 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:18.859221 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:18.860278 21328 raft_consensus.cc:1226] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
I20260812 19:12:18.857076 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:18.868042 21378 ts_tablet_manager.cc:1428] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.089s user 0.009s sys 0.000s
I20260812 19:12:18.878458 21378 tablet_bootstrap.cc:492] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:18.880937 21494 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:18.882257 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:18.883471 21318 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:18.890515 21615 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:18.891580 21615 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:18.894331 21379 tablet_bootstrap.cc:654] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:18.896982 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:18.907094 21239 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:18.919696 21381 tablet_bootstrap.cc:492] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:18.932816 21320 raft_consensus.cc:2454] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
W20260812 19:12:18.938603 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 88684.1us
I20260812 19:12:18.940671 21614 raft_consensus.cc:2781] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260812 19:12:18.940542 21287 leader_election.cc:341] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:18.941259 21614 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:18.941443 21287 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:18.941696 21614 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
W20260812 19:12:18.943625 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 104052us
I20260812 19:12:18.945293 21494 raft_consensus.cc:2726] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:18.945526 21595 raft_consensus.cc:2781] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:18.948787 21610 consensus_queue.cc:1035] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260812 19:12:18.949115 21595 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:18.956921 21595 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:18.957978 21200 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:18.960626 21495 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:18.961293 21495 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:18.961755 21495 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
W20260812 19:12:18.964339 21203 consensus_peers.cc:487] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa -> Peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Couldn't send request to peer f741102bfe1441d1b453e5a838649059. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:18.970791 21381 ts_tablet_manager.cc:1397] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.658s user 0.045s sys 0.052s
W20260812 19:12:18.965881 21203 leader_election.cc:341] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
W20260812 19:12:18.973519 21287 leader_election.cc:341] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:19.015268 21610 consensus_queue.cc:1035] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260812 19:12:19.000291 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 97681.8us
I20260812 19:12:19.000341 21200 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:19.044101 21607 raft_consensus.cc:2726] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260812 19:12:19.045554 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 68199.5us
I20260812 19:12:19.064425 21615 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:12:19.066257 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 52985.2us
I20260812 19:12:19.079778 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:12:19.067329 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 100394us
I20260812 19:12:19.082855 21614 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.089016 21378 tablet_bootstrap.cc:654] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:19.082831 21379 tablet_bootstrap.cc:492] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
W20260812 19:12:19.105845 21287 leader_election.cc:341] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
W20260812 19:12:19.081163 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 62671.7us
I20260812 19:12:19.109463 21287 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:19.099593 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:19.118693 21610 raft_consensus.cc:2726] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:19.099004 21385 consensus_queue.cc:1035] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20260812 19:12:19.122581 21614 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:19.098408 21570 consensus_queue.cc:1035] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260812 19:12:19.124706 21595 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.126834 21595 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:19.130818 21494 consensus_queue.cc:1035] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:19.134262 21379 ts_tablet_manager.cc:1397] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.561s user 0.017s sys 0.025s
W20260812 19:12:19.137321 21093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260812 19:12:19.132776 21381 raft_consensus.cc:348] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.146159 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:19.147306 21153 raft_consensus.cc:2379] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 1: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:12:19.147850 21630 log.cc:924] Time spent T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20260812 19:12:19.146168 21381 raft_consensus.cc:374] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:19.148535 21381 raft_consensus.cc:729] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:19.097435 21495 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.150738 21379 raft_consensus.cc:348] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.151592 21379 raft_consensus.cc:374] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:19.149482 21381 consensus_queue.cc:260] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.151889 21379 raft_consensus.cc:729] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:19.152608 21379 consensus_queue.cc:260] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.153023 21495 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:19.160394 21381 ts_tablet_manager.cc:1428] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.186s user 0.004s sys 0.012s
I20260812 19:12:19.169788 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:19.182741 21614 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:19.183616 21614 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.185933 21614 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:19.187836 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:19.188894 21157 raft_consensus.cc:2379] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 1: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:12:19.174494 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:19.190146 21155 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:19.192023 21112 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:19.193239 21564 consensus_queue.cc:1046] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.005s
I20260812 19:12:19.199889 21564 raft_consensus.cc:3032] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:19.200224 21379 ts_tablet_manager.cc:1428] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.065s user 0.006s sys 0.002s
I20260812 19:12:19.200507 21564 raft_consensus.cc:729] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:19.201668 21379 tablet_bootstrap.cc:492] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:19.202823 21628 raft_consensus.cc:2781] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:19.203435 21628 raft_consensus.cc:686] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:19.198549 21381 tablet_bootstrap.cc:492] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:19.201804 21564 consensus_queue.cc:260] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.210037 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:19.210987 21244 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:19.211064 21564 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:19.204493 21628 consensus_queue.cc:237] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.215706 21203 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:19.216711 21495 raft_consensus.cc:2781] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:19.217126 21495 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:19.217496 21495 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:19.239380 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:19.240770 21320 raft_consensus.cc:2379] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 1: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:12:19.242126 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:19.247277 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:19.250942 21287 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:19.252614 21614 raft_consensus.cc:2726] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:12:19.254920 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:19.298051 21287 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:19.265100 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:19.407655 21328 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:19.307269 21035 catalog_manager.cc:5495] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:19.417433 21244 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
W20260812 19:12:19.419155 21200 leader_election.cc:341] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:19.428958 21379 tablet_bootstrap.cc:654] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:19.420022 21443 consensus_queue.cc:1035] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:19.423804 21495 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.440519 21323 raft_consensus.cc:3037] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:19.466257 21614 raft_consensus.cc:2726] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260812 19:12:19.466732 21203 leader_election.cc:341] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
W20260812 19:12:19.467916 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 169995us
W20260812 19:12:19.472539 21645 log.cc:924] Time spent T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20260812 19:12:19.457240 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 152052us
I20260812 19:12:19.474735 21287 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:19.502087 21378 tablet_bootstrap.cc:492] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:19.505283 21378 ts_tablet_manager.cc:1397] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.635s user 0.085s sys 0.090s
I20260812 19:12:19.504165 21494 raft_consensus.cc:2726] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:19.522913 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:19.526175 21157 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:19.533965 21495 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:19.538784 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:19.541142 21378 raft_consensus.cc:348] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.541965 21114 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:12:19.542016 21378 raft_consensus.cc:374] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:19.543151 21378 raft_consensus.cc:729] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:19.543896 21378 consensus_queue.cc:260] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.548368 21419 raft_consensus.cc:2781] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:19.549057 21419 raft_consensus.cc:686] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:12:19.551846 21200 leader_election.cc:341] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:19.550066 21419 consensus_queue.cc:237] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:19.553035 21649 log.cc:924] Time spent T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260812 19:12:19.524848 21328 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:19.566840 21615 consensus_queue.cc:1035] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:19.573601 21157 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:19.618839 21378 ts_tablet_manager.cc:1428] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.113s user 0.009s sys 0.000s
I20260812 19:12:19.622602 21533 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:19.623963 21533 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.644755 21378 tablet_bootstrap.cc:492] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:19.647359 21533 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:12:19.661737 21626 log.cc:924] Time spent T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.111s user 0.009s sys 0.046s
I20260812 19:12:19.663784 21495 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:19.664333 21495 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.664371 21203 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:12:19.665202 21379 tablet_bootstrap.cc:492] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:19.665720 21379 ts_tablet_manager.cc:1397] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.464s user 0.030s sys 0.026s
I20260812 19:12:19.666108 21637 raft_consensus.cc:2781] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:19.666543 21495 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:19.667016 21637 raft_consensus.cc:686] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:19.668628 21379 raft_consensus.cc:348] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.669442 21379 raft_consensus.cc:374] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:19.668790 21637 consensus_queue.cc:237] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.669766 21379 raft_consensus.cc:729] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:19.670481 21379 consensus_queue.cc:260] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.672916 21379 ts_tablet_manager.cc:1428] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 19:12:19.674269 21379 tablet_bootstrap.cc:492] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:19.675591 21200 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:19.663753 21381 tablet_bootstrap.cc:654] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:19.700695 21615 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:19.701299 21615 raft_consensus.cc:686] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:19.702211 21615 consensus_queue.cc:237] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.703141 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:19.705209 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:19.707261 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:19.708350 21328 raft_consensus.cc:2454] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
W20260812 19:12:19.710971 21287 leader_election.cc:341] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
W20260812 19:12:19.712761 21203 leader_election.cc:341] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:19.726212 21378 tablet_bootstrap.cc:654] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:19.729645 21646 consensus_queue.cc:1035] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260812 19:12:19.737811 21512 consensus_queue.cc:1035] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260812 19:12:19.749663 21379 tablet_bootstrap.cc:654] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:19.752897 21200 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:19.770902 21615 raft_consensus.cc:2781] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:19.776225 21615 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:19.777463 21615 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:19.778653 21381 tablet_bootstrap.cc:492] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:19.779233 21381 ts_tablet_manager.cc:1397] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.581s user 0.024s sys 0.029s
I20260812 19:12:19.785131 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:19.770539 21034 catalog_manager.cc:5495] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
W20260812 19:12:19.792097 21287 leader_election.cc:341] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:19.792593 21287 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:19.793922 21533 raft_consensus.cc:2726] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:19.800338 21381 raft_consensus.cc:348] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.801851 21381 raft_consensus.cc:374] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:19.802629 21381 raft_consensus.cc:729] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:19.803540 21381 consensus_queue.cc:260] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.812108 21379 tablet_bootstrap.cc:492] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:19.812959 21379 ts_tablet_manager.cc:1397] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.139s user 0.011s sys 0.007s
I20260812 19:12:19.814015 21381 ts_tablet_manager.cc:1428] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.034s user 0.007s sys 0.000s
I20260812 19:12:19.815367 21381 tablet_bootstrap.cc:492] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:19.816566 21378 tablet_bootstrap.cc:492] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:19.817101 21378 ts_tablet_manager.cc:1397] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.173s user 0.020s sys 0.004s
I20260812 19:12:19.823530 21378 raft_consensus.cc:348] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.824358 21378 raft_consensus.cc:374] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:19.824635 21378 raft_consensus.cc:729] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:19.825265 21378 consensus_queue.cc:260] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.827369 21379 raft_consensus.cc:348] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.829854 21379 raft_consensus.cc:374] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:19.830204 21379 raft_consensus.cc:729] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:19.819517 21615 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.832901 21615 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:19.834936 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:19.835270 21378 ts_tablet_manager.cc:1428] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.018s user 0.007s sys 0.002s
I20260812 19:12:19.835803 21157 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:19.837416 21378 tablet_bootstrap.cc:492] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:19.840755 21381 tablet_bootstrap.cc:654] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:19.842398 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:19.843240 21328 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:19.847461 21157 raft_consensus.cc:2454] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:19.847039 21036 catalog_manager.cc:5495] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:19.835831 21379 consensus_queue.cc:260] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.857262 21203 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:19.863121 21378 tablet_bootstrap.cc:654] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:19.862998 21495 raft_consensus.cc:2781] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:19.863732 21495 raft_consensus.cc:686] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:19.864717 21495 consensus_queue.cc:237] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.872040 21035 catalog_manager.cc:5495] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:19.875029 21615 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:19.875698 21035 catalog_manager.cc:5495] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:19.876540 21615 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.879794 21615 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:19.880436 21379 ts_tablet_manager.cc:1428] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.067s user 0.009s sys 0.002s
I20260812 19:12:19.882090 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:19.885540 21203 leader_election.cc:341] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:19.885800 21381 tablet_bootstrap.cc:492] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:19.886447 21381 ts_tablet_manager.cc:1397] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.071s user 0.016s sys 0.005s
I20260812 19:12:19.881242 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:19.858389 21328 raft_consensus.cc:2454] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:19.888216 21379 tablet_bootstrap.cc:492] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:19.889321 21381 raft_consensus.cc:348] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.890143 21381 raft_consensus.cc:374] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:19.890463 21381 raft_consensus.cc:729] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:19.892545 21381 consensus_queue.cc:260] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.895072 21379 tablet_bootstrap.cc:654] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:19.896148 21378 tablet_bootstrap.cc:492] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:19.896741 21378 ts_tablet_manager.cc:1397] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.060s user 0.017s sys 0.002s
I20260812 19:12:19.898142 21381 ts_tablet_manager.cc:1428] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
W20260812 19:12:19.898522 21200 leader_election.cc:341] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:19.899113 21200 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:19.899971 21615 raft_consensus.cc:2726] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:19.913106 21378 raft_consensus.cc:348] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.914535 21378 raft_consensus.cc:374] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:19.914961 21378 raft_consensus.cc:729] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:19.914192 21381 tablet_bootstrap.cc:492] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:19.919099 21378 consensus_queue.cc:260] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.922406 21378 ts_tablet_manager.cc:1428] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.025s user 0.006s sys 0.001s
I20260812 19:12:19.923993 21378 tablet_bootstrap.cc:492] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:19.924521 21381 tablet_bootstrap.cc:654] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:19.929565 21035 catalog_manager.cc:5495] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:19.935484 21378 tablet_bootstrap.cc:654] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:19.939450 21379 tablet_bootstrap.cc:492] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:19.940028 21379 ts_tablet_manager.cc:1397] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.057s user 0.024s sys 0.013s
I20260812 19:12:19.942880 21379 raft_consensus.cc:348] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.950326 21381 tablet_bootstrap.cc:492] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:19.953017 21379 raft_consensus.cc:374] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:19.953269 21381 ts_tablet_manager.cc:1397] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.054s user 0.013s sys 0.009s
I20260812 19:12:19.953356 21379 raft_consensus.cc:729] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:19.954140 21379 consensus_queue.cc:260] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.956444 21379 ts_tablet_manager.cc:1428] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260812 19:12:19.956089 21381 raft_consensus.cc:348] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.957629 21379 tablet_bootstrap.cc:492] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:19.957547 21381 raft_consensus.cc:374] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:19.958164 21381 raft_consensus.cc:729] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:19.959043 21381 consensus_queue.cc:260] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.970054 21381 ts_tablet_manager.cc:1428] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260812 19:12:19.977453 21381 tablet_bootstrap.cc:492] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:19.982002 21378 tablet_bootstrap.cc:492] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:19.982645 21378 ts_tablet_manager.cc:1397] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.059s user 0.014s sys 0.005s
I20260812 19:12:19.985193 21378 raft_consensus.cc:348] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.985942 21378 raft_consensus.cc:374] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:19.986188 21378 raft_consensus.cc:729] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:19.986900 21378 consensus_queue.cc:260] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:19.989156 21378 ts_tablet_manager.cc:1428] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20260812 19:12:19.990324 21378 tablet_bootstrap.cc:492] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:19.992100 21381 tablet_bootstrap.cc:654] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:19.996802 21378 tablet_bootstrap.cc:654] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:19.972155 21379 tablet_bootstrap.cc:654] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:20.008270 21381 tablet_bootstrap.cc:492] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:20.009243 21381 ts_tablet_manager.cc:1397] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.000s
I20260812 19:12:20.009712 21379 tablet_bootstrap.cc:492] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:20.010275 21379 ts_tablet_manager.cc:1397] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.053s user 0.014s sys 0.000s
I20260812 19:12:20.013337 21379 raft_consensus.cc:348] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.014127 21379 raft_consensus.cc:374] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:20.016870 21378 tablet_bootstrap.cc:492] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:20.017468 21378 ts_tablet_manager.cc:1397] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.000s
I20260812 19:12:20.020304 21378 raft_consensus.cc:348] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.021088 21378 raft_consensus.cc:374] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:20.021379 21378 raft_consensus.cc:729] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:20.025017 21381 raft_consensus.cc:348] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.014396 21379 raft_consensus.cc:729] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:20.025988 21381 raft_consensus.cc:374] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:20.026501 21381 raft_consensus.cc:729] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:20.026957 21379 consensus_queue.cc:260] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.030084 21379 ts_tablet_manager.cc:1428] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.020s user 0.009s sys 0.000s
I20260812 19:12:20.030964 21378 consensus_queue.cc:260] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.027534 21381 consensus_queue.cc:260] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.035401 21381 ts_tablet_manager.cc:1428] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20260812 19:12:20.036681 21381 tablet_bootstrap.cc:492] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:20.040307 21379 tablet_bootstrap.cc:492] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:20.042852 21378 ts_tablet_manager.cc:1428] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.025s user 0.009s sys 0.000s
I20260812 19:12:20.044188 21615 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:20.045135 21378 tablet_bootstrap.cc:492] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:20.044746 21615 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.048318 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:20.048830 21381 tablet_bootstrap.cc:654] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:20.049120 21319 raft_consensus.cc:2454] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:20.050678 21200 leader_election.cc:302] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:20.051645 21385 raft_consensus.cc:2781] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:20.052037 21385 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:20.052446 21385 raft_consensus.cc:3037] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:20.052753 21378 tablet_bootstrap.cc:654] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:20.052145 21564 consensus_queue.cc:1035] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:20.056093 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:20.057138 21153 raft_consensus.cc:2454] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:20.058903 21379 tablet_bootstrap.cc:654] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:20.063138 21381 tablet_bootstrap.cc:492] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:20.063679 21381 ts_tablet_manager.cc:1397] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.005s
I20260812 19:12:20.066371 21381 raft_consensus.cc:348] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.067193 21381 raft_consensus.cc:374] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:20.067513 21381 raft_consensus.cc:729] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:20.067610 21615 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:20.068586 21381 consensus_queue.cc:260] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.074213 21381 ts_tablet_manager.cc:1428] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260812 19:12:20.075357 21381 tablet_bootstrap.cc:492] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:20.081817 21381 tablet_bootstrap.cc:654] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:20.085477 21512 consensus_queue.cc:1035] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:20.085280 21385 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.113540 21533 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:20.114354 21533 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.105805 21381 tablet_bootstrap.cc:492] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:20.117650 21381 ts_tablet_manager.cc:1397] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.019s
I20260812 19:12:20.120302 21381 raft_consensus.cc:348] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.121059 21381 raft_consensus.cc:374] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:20.121342 21381 raft_consensus.cc:729] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:20.123255 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:20.125797 21287 leader_election.cc:341] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:20.129341 21385 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:20.129705 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:20.130648 21319 raft_consensus.cc:3037] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:20.131681 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:20.132516 21157 raft_consensus.cc:3037] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:20.121989 21381 consensus_queue.cc:260] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.136152 21381 ts_tablet_manager.cc:1428] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260812 19:12:20.137315 21381 tablet_bootstrap.cc:492] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:20.140172 21319 raft_consensus.cc:2454] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:20.141778 21200 leader_election.cc:302] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:20.143275 21615 raft_consensus.cc:2781] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:20.143886 21615 raft_consensus.cc:686] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:20.145260 21381 tablet_bootstrap.cc:654] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:20.144991 21615 consensus_queue.cc:237] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.149820 21379 tablet_bootstrap.cc:492] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:20.150372 21379 ts_tablet_manager.cc:1397] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.110s user 0.015s sys 0.024s
I20260812 19:12:20.152071 21153 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:20.152554 21533 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:20.153257 21379 raft_consensus.cc:348] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.154153 21379 raft_consensus.cc:374] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:20.154500 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:12:20.156368 21287 leader_election.cc:341] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:20.154481 21379 raft_consensus.cc:729] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:20.156908 21287 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:20.158077 21379 consensus_queue.cc:260] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.158742 21494 raft_consensus.cc:2726] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:20.170001 21385 consensus_queue.cc:1035] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260812 19:12:20.183193 21683 log.cc:924] Time spent T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260812 19:12:20.156502 21378 tablet_bootstrap.cc:492] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:20.184110 21378 ts_tablet_manager.cc:1397] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.139s user 0.024s sys 0.017s
I20260812 19:12:20.185477 21385 consensus_queue.cc:1035] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:20.197280 21533 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:20.194214 21378 raft_consensus.cc:348] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.202780 21378 raft_consensus.cc:374] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:20.203181 21378 raft_consensus.cc:729] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:20.197962 21533 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.209393 21155 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:20.211305 21378 consensus_queue.cc:260] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.206476 21533 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:20.217630 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:20.218272 21157 raft_consensus.cc:2454] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:20.219132 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:20.223132 21378 ts_tablet_manager.cc:1428] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.039s user 0.007s sys 0.002s
I20260812 19:12:20.229642 21378 tablet_bootstrap.cc:492] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:20.230839 21379 ts_tablet_manager.cc:1428] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.080s user 0.018s sys 0.018s
I20260812 19:12:20.238161 21287 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:20.242481 21328 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:20.256086 21381 tablet_bootstrap.cc:492] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:20.256639 21381 ts_tablet_manager.cc:1397] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.120s user 0.015s sys 0.028s
I20260812 19:12:20.257078 21379 tablet_bootstrap.cc:492] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:20.267307 21381 raft_consensus.cc:348] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.268376 21381 raft_consensus.cc:374] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:20.268716 21381 raft_consensus.cc:729] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:20.269567 21381 consensus_queue.cc:260] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.272017 21533 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:20.282893 21385 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:20.294039 21533 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.295473 21385 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.298651 21533 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:20.299823 21385 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:20.303170 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:20.307242 21203 leader_election.cc:341] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:20.308369 21495 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:20.308981 21495 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.310123 21379 tablet_bootstrap.cc:654] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:20.311870 21381 ts_tablet_manager.cc:1428] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.055s user 0.004s sys 0.006s
I20260812 19:12:20.292174 21378 tablet_bootstrap.cc:654] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:20.315740 21495 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:20.316066 21685 raft_consensus.cc:2726] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:20.317196 21381 tablet_bootstrap.cc:492] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:20.311777 21035 catalog_manager.cc:5495] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:20.336874 21564 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:20.337697 21564 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.323400 21646 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:20.342589 21646 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.325114 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:20.348891 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:20.348747 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:20.349653 21323 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:20.340083 21564 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:20.352705 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:20.353636 21328 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:20.355541 21112 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:20.356611 21419 raft_consensus.cc:2781] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:20.356994 21419 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:20.357342 21419 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:20.359100 21646 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:20.354277 21564 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:20.363874 21564 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.364044 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:20.372078 21637 consensus_queue.cc:1035] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260812 19:12:20.374969 21287 leader_election.cc:341] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:20.365407 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:20.384441 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:12:20.385095 21200 leader_election.cc:341] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:20.385269 21244 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:20.385614 21200 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:20.387132 21200 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:20.390144 21287 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:20.390012 21385 raft_consensus.cc:2781] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:20.387698 21319 raft_consensus.cc:2454] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:20.394418 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:20.395321 21237 raft_consensus.cc:2454] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:20.385280 21495 consensus_queue.cc:1035] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260812 19:12:20.388351 21570 raft_consensus.cc:2726] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:20.394361 21385 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:20.396510 21685 raft_consensus.cc:2781] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:20.424587 21685 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:20.425068 21685 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
W20260812 19:12:20.399751 21114 leader_election.cc:341] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:20.429610 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:20.431247 21385 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:20.401433 21112 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:20.421953 21564 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:20.397290 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:20.436002 21512 raft_consensus.cc:2781] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:20.436892 21512 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:20.437398 21512 raft_consensus.cc:3037] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
W20260812 19:12:20.439661 21203 leader_election.cc:341] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
W20260812 19:12:20.456427 21093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260812 19:12:20.457257 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:20.422089 21419 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.458052 21320 raft_consensus.cc:2454] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:20.461153 21419 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:20.463729 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:20.464524 21320 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
W20260812 19:12:20.450436 21114 leader_election.cc:341] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:20.465683 21512 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.466463 21112 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:20.467362 21419 raft_consensus.cc:2781] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:20.467747 21419 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:20.468098 21419 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:20.469000 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:20.472378 21512 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:20.472939 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:20.473632 21319 raft_consensus.cc:3037] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:20.477789 21495 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:12:20.472416 21381 tablet_bootstrap.cc:654] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:20.445652 21615 consensus_queue.cc:1035] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20260812 19:12:20.478420 21495 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.500909 21691 consensus_queue.cc:1035] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260812 19:12:20.505311 21320 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:20.515156 21495 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:20.481927 21385 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.529557 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:20.520030 21419 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.530364 21155 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:12:20.531705 21419 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:20.519207 21112 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:20.533102 21628 raft_consensus.cc:2781] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:20.533883 21628 raft_consensus.cc:686] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:12:20.547375 21114 leader_election.cc:341] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:20.537664 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:20.537186 21628 consensus_queue.cc:237] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.593590 21378 tablet_bootstrap.cc:492] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:20.594326 21378 ts_tablet_manager.cc:1397] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.365s user 0.017s sys 0.008s
I20260812 19:12:20.506757 21685 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.590191 21239 raft_consensus.cc:3037] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:20.601552 21385 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:20.540577 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:20.603902 21319 raft_consensus.cc:2454] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:20.601928 21379 tablet_bootstrap.cc:492] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:20.606851 21379 ts_tablet_manager.cc:1397] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.362s user 0.025s sys 0.009s
W20260812 19:12:20.590583 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 51558.6us
I20260812 19:12:20.607081 21329 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:20.612593 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:20.596699 21203 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:20.619199 21495 raft_consensus.cc:2781] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:12:20.619750 21495 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:12:20.620048 21637 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:20.620170 21495 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:20.620620 21637 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.622977 21329 raft_consensus.cc:2454] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:20.624338 21112 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:20.626811 21512 raft_consensus.cc:2781] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election won for term 1
W20260812 19:12:20.626292 21114 leader_election.cc:341] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:20.629621 21379 raft_consensus.cc:348] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.631271 21379 raft_consensus.cc:374] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:20.631773 21112 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:20.632030 21379 raft_consensus.cc:729] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:20.633379 21379 consensus_queue.cc:260] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.637833 21564 raft_consensus.cc:2781] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:20.652542 21379 ts_tablet_manager.cc:1428] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.045s user 0.008s sys 0.002s
I20260812 19:12:20.660142 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:20.665755 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:20.673697 21378 raft_consensus.cc:348] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.657464 21564 raft_consensus.cc:686] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:20.683658 21564 consensus_queue.cc:237] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:20.692538 21203 leader_election.cc:341] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
W20260812 19:12:20.695446 21203 leader_election.cc:341] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:20.647387 21637 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:20.702759 21378 raft_consensus.cc:374] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:20.703496 21378 raft_consensus.cc:729] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:20.704667 21378 consensus_queue.cc:260] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.707134 21685 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:20.689852 21495 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.711723 21495 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:20.708344 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:12:20.713522 21697 log.cc:924] Time spent T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20260812 19:12:20.713932 21244 raft_consensus.cc:2379] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 1: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:12:20.720474 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:12:20.722944 21287 leader_election.cc:341] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:20.723606 21287 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:20.726540 21685 raft_consensus.cc:2726] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260812 19:12:20.727689 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:20.728641 21328 raft_consensus.cc:2379] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 2: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:12:20.729625 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:20.729532 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:20.731114 21320 raft_consensus.cc:2379] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 1: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:12:20.746956 21381 tablet_bootstrap.cc:492] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:20.747234 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:20.747748 21381 ts_tablet_manager.cc:1397] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.431s user 0.015s sys 0.007s
I20260812 19:12:20.748248 21329 raft_consensus.cc:2379] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 2: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:12:20.742928 21512 raft_consensus.cc:686] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:20.750270 21239 raft_consensus.cc:2454] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:20.749836 21512 consensus_queue.cc:237] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.746502 21378 ts_tablet_manager.cc:1428] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.152s user 0.007s sys 0.001s
I20260812 19:12:20.752173 21381 raft_consensus.cc:348] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.753325 21381 raft_consensus.cc:374] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:20.753702 21381 raft_consensus.cc:729] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:20.755664 21381 consensus_queue.cc:260] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.759385 21381 ts_tablet_manager.cc:1428] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20260812 19:12:20.760963 21200 leader_election.cc:341] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:20.765939 21200 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:20.765247 21378 tablet_bootstrap.cc:492] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:20.768028 21379 tablet_bootstrap.cc:492] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:20.770956 21200 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:20.772084 21381 tablet_bootstrap.cc:492] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:20.773237 21035 catalog_manager.cc:5495] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:20.774384 21570 raft_consensus.cc:2726] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260812 19:12:20.776072 21035 catalog_manager.cc:5495] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:20.775838 21495 raft_consensus.cc:2726] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:20.779994 21696 consensus_queue.cc:1035] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:20.779783 21035 catalog_manager.cc:5495] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:20.784066 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:20.808640 21637 consensus_queue.cc:1035] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260812 19:12:20.816529 21153 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:12:20.827261 21203 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:12:20.828780 21615 raft_consensus.cc:2781] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:20.829603 21615 raft_consensus.cc:686] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:20.831035 21615 consensus_queue.cc:237] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.829015 21379 tablet_bootstrap.cc:654] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:20.851457 21378 tablet_bootstrap.cc:654] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:20.871948 21034 catalog_manager.cc:5495] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 1 to 2, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 2 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:20.876317 21381 tablet_bootstrap.cc:654] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:20.889972 21378 tablet_bootstrap.cc:492] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:20.893620 21378 ts_tablet_manager.cc:1397] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.129s user 0.013s sys 0.003s
I20260812 19:12:20.894591 21379 tablet_bootstrap.cc:492] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:20.895162 21379 ts_tablet_manager.cc:1397] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.153s user 0.018s sys 0.008s
I20260812 19:12:20.897831 21378 raft_consensus.cc:348] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.898689 21378 raft_consensus.cc:374] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:20.898949 21378 raft_consensus.cc:729] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:20.899590 21378 consensus_queue.cc:260] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.904388 21379 raft_consensus.cc:348] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.905221 21379 raft_consensus.cc:374] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:20.905511 21379 raft_consensus.cc:729] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:20.907298 21378 ts_tablet_manager.cc:1428] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260812 19:12:20.908470 21378 tablet_bootstrap.cc:492] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:20.913717 21615 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:20.914652 21615 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.917200 21615 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:20.920068 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:20.919910 21379 consensus_queue.cc:260] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:20.922026 21200 leader_election.cc:341] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:20.923090 21379 ts_tablet_manager.cc:1428] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.028s user 0.008s sys 0.001s
I20260812 19:12:20.926968 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:20.928820 21379 tablet_bootstrap.cc:492] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
W20260812 19:12:20.932588 21203 leader_election.cc:341] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:20.933172 21203 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:20.933992 21615 raft_consensus.cc:2726] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:20.936838 21379 tablet_bootstrap.cc:654] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:20.937820 21381 tablet_bootstrap.cc:492] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:20.938289 21381 ts_tablet_manager.cc:1397] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.166s user 0.010s sys 0.012s
I20260812 19:12:20.942090 21378 tablet_bootstrap.cc:654] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:20.941835 21381 raft_consensus.cc:348] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.942590 21381 raft_consensus.cc:374] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:20.966233 21381 raft_consensus.cc:729] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:20.972393 21381 consensus_queue.cc:260] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.972891 21512 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:20.973650 21512 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:20.977622 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:20.978701 21244 raft_consensus.cc:2379] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 1: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:12:20.980819 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:20.981851 21329 raft_consensus.cc:2379] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 1: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:12:20.983031 21112 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:20.984498 21564 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:20.986683 21379 tablet_bootstrap.cc:492] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:20.987234 21379 ts_tablet_manager.cc:1397] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.020s
I20260812 19:12:20.987540 21381 ts_tablet_manager.cc:1428] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.049s user 0.003s sys 0.007s
I20260812 19:12:20.991400 21512 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:20.993597 21381 tablet_bootstrap.cc:492] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:20.997361 21564 raft_consensus.cc:2726] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:20.999382 21379 raft_consensus.cc:348] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.000581 21379 raft_consensus.cc:374] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.000957 21379 raft_consensus.cc:729] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.002375 21378 tablet_bootstrap.cc:492] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:21.002930 21378 ts_tablet_manager.cc:1397] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.095s user 0.012s sys 0.005s
I20260812 19:12:21.002586 21379 consensus_queue.cc:260] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.009119 21378 raft_consensus.cc:348] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.010572 21378 raft_consensus.cc:374] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.010968 21378 raft_consensus.cc:729] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.011650 21378 consensus_queue.cc:260] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.013432 21379 ts_tablet_manager.cc:1428] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.026s user 0.006s sys 0.003s
I20260812 19:12:21.017275 21378 ts_tablet_manager.cc:1428] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.014s user 0.002s sys 0.004s
I20260812 19:12:21.017917 21379 tablet_bootstrap.cc:492] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:21.018390 21378 tablet_bootstrap.cc:492] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:21.023029 21381 tablet_bootstrap.cc:654] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.025473 21244 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:21.026746 21378 tablet_bootstrap.cc:654] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.038259 21564 consensus_queue.cc:1035] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:21.043617 21379 tablet_bootstrap.cc:654] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.051029 21512 consensus_queue.cc:1035] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:21.059370 21381 tablet_bootstrap.cc:492] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:21.059949 21381 ts_tablet_manager.cc:1397] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.067s user 0.014s sys 0.003s
I20260812 19:12:21.062912 21381 raft_consensus.cc:348] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.063690 21381 raft_consensus.cc:374] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.064611 21378 tablet_bootstrap.cc:492] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:21.063956 21381 raft_consensus.cc:729] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.065135 21378 ts_tablet_manager.cc:1397] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.003s
I20260812 19:12:21.065376 21381 consensus_queue.cc:260] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.068163 21378 raft_consensus.cc:348] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.069105 21378 raft_consensus.cc:374] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.069360 21378 raft_consensus.cc:729] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.069978 21378 consensus_queue.cc:260] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.068125 21381 ts_tablet_manager.cc:1428] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260812 19:12:21.072408 21378 ts_tablet_manager.cc:1428] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 19:12:21.074602 21378 tablet_bootstrap.cc:492] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:21.077386 21495 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:21.078048 21495 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.081079 21381 tablet_bootstrap.cc:492] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:21.082806 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:21.083547 21158 raft_consensus.cc:2454] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:21.085040 21203 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:21.086055 21696 raft_consensus.cc:2781] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:21.086428 21696 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:21.086853 21696 raft_consensus.cc:3037] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:21.088435 21379 tablet_bootstrap.cc:492] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:21.089052 21379 ts_tablet_manager.cc:1397] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.071s user 0.016s sys 0.008s
I20260812 19:12:21.089973 21381 tablet_bootstrap.cc:654] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.091773 21379 raft_consensus.cc:348] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.092655 21379 raft_consensus.cc:374] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.092945 21379 raft_consensus.cc:729] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.093621 21379 consensus_queue.cc:260] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.094331 21378 tablet_bootstrap.cc:654] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.096625 21379 ts_tablet_manager.cc:1428] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260812 19:12:21.097858 21379 tablet_bootstrap.cc:492] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:21.107916 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:21.108784 21329 raft_consensus.cc:2454] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:21.111334 21494 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:21.111866 21494 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.114591 21494 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:21.115119 21494 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:21.115538 21494 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.115837 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:21.116894 21158 raft_consensus.cc:2454] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:21.117650 21494 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:21.118366 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:21.118935 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:21.125900 21244 raft_consensus.cc:2454] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:21.119999 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:21.126888 21239 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:21.128629 21287 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:21.125041 21495 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:21.129616 21685 raft_consensus.cc:2781] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:21.131433 21685 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:21.132076 21685 raft_consensus.cc:3037] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:21.134506 21287 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:12:21.135552 21287 leader_election.cc:341] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:21.135849 21381 tablet_bootstrap.cc:492] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:21.136397 21381 ts_tablet_manager.cc:1397] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.056s user 0.013s sys 0.011s
I20260812 19:12:21.139370 21381 raft_consensus.cc:348] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.139746 21735 raft_consensus.cc:2781] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:21.140455 21381 raft_consensus.cc:374] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.140892 21381 raft_consensus.cc:729] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.141024 21735 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:21.139707 21512 consensus_queue.cc:1035] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260812 19:12:21.141932 21735 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:21.139655 21696 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.151578 21378 tablet_bootstrap.cc:492] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:21.152125 21378 ts_tablet_manager.cc:1397] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.078s user 0.024s sys 0.013s
I20260812 19:12:21.141681 21381 consensus_queue.cc:260] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.145272 21379 tablet_bootstrap.cc:654] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.161170 21381 ts_tablet_manager.cc:1428] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260812 19:12:21.162676 21378 raft_consensus.cc:348] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.163481 21378 raft_consensus.cc:374] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.163745 21378 raft_consensus.cc:729] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.164201 21512 consensus_queue.cc:1035] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:21.164377 21378 consensus_queue.cc:260] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.166649 21378 ts_tablet_manager.cc:1428] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260812 19:12:21.170303 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:21.175544 21378 tablet_bootstrap.cc:492] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:21.167313 21685 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.171079 21158 raft_consensus.cc:3037] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:21.186718 21381 tablet_bootstrap.cc:492] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:21.184151 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:21.191709 21685 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:21.192880 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:21.193555 21237 raft_consensus.cc:3037] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:21.195421 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:21.196151 21320 raft_consensus.cc:3037] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:21.189042 21696 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:21.191412 21155 raft_consensus.cc:3037] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:21.208606 21379 tablet_bootstrap.cc:492] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:21.210533 21379 ts_tablet_manager.cc:1397] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.113s user 0.005s sys 0.024s
I20260812 19:12:21.215292 21735 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.218384 21735 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:21.221522 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:21.222098 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:21.222262 21239 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
W20260812 19:12:21.223780 21287 leader_election.cc:341] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:21.230834 21320 raft_consensus.cc:2454] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:21.232479 21200 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:21.233968 21570 raft_consensus.cc:2781] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:21.234130 21158 raft_consensus.cc:2454] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:21.234586 21570 raft_consensus.cc:686] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:21.237610 21155 raft_consensus.cc:2454] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:21.238409 21378 tablet_bootstrap.cc:654] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.238880 21287 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:21.239532 21237 raft_consensus.cc:2454] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:21.240149 21570 consensus_queue.cc:237] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.244227 21239 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:21.245466 21287 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:21.243265 21735 raft_consensus.cc:2781] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:21.246892 21734 raft_consensus.cc:2781] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:21.247179 21735 raft_consensus.cc:686] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:21.247514 21734 raft_consensus.cc:686] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:21.249212 21734 consensus_queue.cc:237] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.254174 21379 raft_consensus.cc:348] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.255607 21379 raft_consensus.cc:374] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.256502 21685 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:21.257370 21685 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.255997 21379 raft_consensus.cc:729] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.259306 21735 consensus_queue.cc:237] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.261293 21381 tablet_bootstrap.cc:654] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.260672 21379 consensus_queue.cc:260] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.298714 21494 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:21.299818 21696 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:21.299530 21494 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.304446 21696 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.315495 21685 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:21.316581 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:21.317164 21696 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:21.317983 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:21.318847 21155 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:21.319666 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:21.321218 21378 tablet_bootstrap.cc:492] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:21.321352 21287 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:21.323654 21378 ts_tablet_manager.cc:1397] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.151s user 0.020s sys 0.009s
W20260812 19:12:21.323511 21200 leader_election.cc:341] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:21.324206 21734 raft_consensus.cc:2781] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:12:21.328718 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:21.330188 21155 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:21.331746 21203 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:12:21.333542 21637 raft_consensus.cc:2781] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:21.334184 21637 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:21.335958 21379 ts_tablet_manager.cc:1428] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.125s user 0.009s sys 0.010s
I20260812 19:12:21.336097 21494 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:21.336256 21637 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:21.343398 21155 raft_consensus.cc:1261] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:21.324735 21734 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:21.345151 21378 raft_consensus.cc:348] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.346000 21378 raft_consensus.cc:374] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.349392 21378 raft_consensus.cc:729] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.350091 21378 consensus_queue.cc:260] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.352279 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:21.352166 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:12:21.354065 21287 leader_election.cc:341] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:21.346340 21734 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:21.356231 21034 catalog_manager.cc:5495] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:21.356452 21615 consensus_queue.cc:1035] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260812 19:12:21.367750 21685 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:21.368832 21685 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:21.385659 21287 leader_election.cc:341] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:21.386183 21287 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:21.374887 21685 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:21.390455 21381 tablet_bootstrap.cc:492] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:21.391459 21381 ts_tablet_manager.cc:1397] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.217s user 0.013s sys 0.006s
I20260812 19:12:21.388746 21745 raft_consensus.cc:2726] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:21.394163 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:21.394999 21237 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:21.394948 21239 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:21.398326 21287 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:21.386752 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:21.400926 21381 raft_consensus.cc:348] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.403155 21381 raft_consensus.cc:374] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.403546 21381 raft_consensus.cc:729] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.404488 21381 consensus_queue.cc:260] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.401433 21745 raft_consensus.cc:2781] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:21.406638 21745 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:21.407148 21745 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:21.417575 21379 tablet_bootstrap.cc:492] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
W20260812 19:12:21.424304 21287 leader_election.cc:341] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:21.434801 21381 ts_tablet_manager.cc:1428] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.043s user 0.008s sys 0.002s
I20260812 19:12:21.419169 21734 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.436985 21419 consensus_queue.cc:1035] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260812 19:12:21.445710 21637 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.447489 21379 tablet_bootstrap.cc:654] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.448712 21378 ts_tablet_manager.cc:1428] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.113s user 0.012s sys 0.035s
I20260812 19:12:21.453920 21381 tablet_bootstrap.cc:492] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:21.460789 21319 raft_consensus.cc:1226] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: []
I20260812 19:12:21.465476 21747 consensus_queue.cc:1046] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:21.466792 21570 raft_consensus.cc:3032] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:12:21.466951 21734 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:21.467317 21570 raft_consensus.cc:729] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:21.468263 21378 tablet_bootstrap.cc:492] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:21.455019 21036 catalog_manager.cc:5495] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:21.472026 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:21.471619 21570 consensus_queue.cc:260] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.475199 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:12:21.477082 21200 leader_election.cc:341] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:21.471136 21036 catalog_manager.cc:5495] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:21.467860 21637 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:21.475643 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:21.457792 21745 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.475572 21570 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:21.475137 21155 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:21.493690 21735 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:21.494298 21735 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.496836 21745 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:21.490876 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:21.500551 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:21.502820 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:21.503559 21240 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
W20260812 19:12:21.506374 21287 leader_election.cc:341] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:21.499714 21735 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:21.511792 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:21.521983 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:12:21.523159 21287 leader_election.cc:341] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:21.523689 21287 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:21.525748 21494 raft_consensus.cc:2726] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260812 19:12:21.526979 21287 leader_election.cc:341] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:21.535287 21628 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:21.535876 21628 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.538085 21628 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:21.539608 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:21.541648 21381 tablet_bootstrap.cc:654] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.542321 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:21.543778 21244 raft_consensus.cc:2379] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 1: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
W20260812 19:12:21.544569 21749 log.cc:924] Time spent T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.050s user 0.000s sys 0.001s
I20260812 19:12:21.545094 21114 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:21.550379 21419 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:21.551672 21379 tablet_bootstrap.cc:492] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:21.552376 21379 ts_tablet_manager.cc:1397] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.135s user 0.012s sys 0.021s
I20260812 19:12:21.555907 21379 raft_consensus.cc:348] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.557003 21379 raft_consensus.cc:374] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.557322 21379 raft_consensus.cc:729] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.558447 21379 consensus_queue.cc:260] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.560786 21495 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:21.561245 21495 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.566850 21495 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:21.570967 21240 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:21.573019 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:21.571728 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:21.575906 21379 ts_tablet_manager.cc:1428] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20260812 19:12:21.585645 21379 tablet_bootstrap.cc:492] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:21.583910 21378 tablet_bootstrap.cc:654] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.576992 21628 consensus_queue.cc:1035] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260812 19:12:21.579051 21287 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:21.593001 21419 raft_consensus.cc:2726] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:21.593817 21735 raft_consensus.cc:2781] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:21.594529 21735 raft_consensus.cc:686] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:21.595805 21735 consensus_queue.cc:237] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.598790 21155 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:21.599293 21239 raft_consensus.cc:2396] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260812 19:12:21.577925 21200 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:21.601156 21203 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:12:21.602715 21747 raft_consensus.cc:2781] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:21.607074 21747 raft_consensus.cc:686] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:21.612500 21287 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:21.617995 21381 tablet_bootstrap.cc:492] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:21.618644 21381 ts_tablet_manager.cc:1397] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.165s user 0.027s sys 0.006s
I20260812 19:12:21.618021 21734 raft_consensus.cc:2726] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:12:21.608881 21747 consensus_queue.cc:237] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.625212 21495 raft_consensus.cc:2726] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:12:21.627131 21381 raft_consensus.cc:348] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.627960 21381 raft_consensus.cc:374] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.628296 21381 raft_consensus.cc:729] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.628958 21381 consensus_queue.cc:260] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.655352 21381 ts_tablet_manager.cc:1428] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.036s user 0.008s sys 0.000s
I20260812 19:12:21.656451 21381 tablet_bootstrap.cc:492] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:21.661860 21378 tablet_bootstrap.cc:492] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:21.662045 21495 consensus_queue.cc:1035] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:12:21.662668 21378 ts_tablet_manager.cc:1397] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.195s user 0.012s sys 0.016s
I20260812 19:12:21.665982 21381 tablet_bootstrap.cc:654] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.666074 21378 raft_consensus.cc:348] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.667106 21378 raft_consensus.cc:374] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.667469 21378 raft_consensus.cc:729] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.668447 21378 consensus_queue.cc:260] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.680335 21378 ts_tablet_manager.cc:1428] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.017s user 0.006s sys 0.002s
I20260812 19:12:21.681530 21378 tablet_bootstrap.cc:492] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:21.681748 21735 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:21.682212 21735 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.684644 21735 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:21.686831 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:21.688642 21287 leader_election.cc:341] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:21.689189 21378 tablet_bootstrap.cc:654] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.694671 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:12:21.696046 21287 leader_election.cc:341] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:21.696481 21287 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:21.697228 21381 tablet_bootstrap.cc:492] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:21.697247 21735 raft_consensus.cc:2726] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:21.697878 21381 ts_tablet_manager.cc:1397] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.025s
I20260812 19:12:21.701606 21381 raft_consensus.cc:348] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:21.703689 21093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260812 19:12:21.709405 21036 catalog_manager.cc:5495] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:21.711514 21378 tablet_bootstrap.cc:492] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:21.712016 21378 ts_tablet_manager.cc:1397] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.001s
I20260812 19:12:21.655354 21379 tablet_bootstrap.cc:654] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.714393 21378 raft_consensus.cc:348] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.715186 21378 raft_consensus.cc:374] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.715462 21378 raft_consensus.cc:729] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.716105 21378 consensus_queue.cc:260] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.719319 21378 ts_tablet_manager.cc:1428] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260812 19:12:21.720355 21378 tablet_bootstrap.cc:492] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:21.725150 21379 tablet_bootstrap.cc:492] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:21.725699 21379 ts_tablet_manager.cc:1397] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.140s user 0.010s sys 0.003s
I20260812 19:12:21.726236 21378 tablet_bootstrap.cc:654] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.728260 21379 raft_consensus.cc:348] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.729048 21379 raft_consensus.cc:374] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.729352 21379 raft_consensus.cc:729] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.731788 21381 raft_consensus.cc:374] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.732146 21381 raft_consensus.cc:729] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.733728 21381 consensus_queue.cc:260] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.737622 21381 ts_tablet_manager.cc:1428] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.039s user 0.004s sys 0.004s
I20260812 19:12:21.738984 21381 tablet_bootstrap.cc:492] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:21.740476 21570 consensus_queue.cc:1035] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:21.741760 21037 catalog_manager.cc:5495] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:21.730015 21379 consensus_queue.cc:260] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.747371 21379 ts_tablet_manager.cc:1428] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20260812 19:12:21.748484 21379 tablet_bootstrap.cc:492] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:21.748940 21378 tablet_bootstrap.cc:492] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:21.749320 21378 ts_tablet_manager.cc:1397] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.004s
I20260812 19:12:21.752182 21378 raft_consensus.cc:348] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.753041 21378 raft_consensus.cc:374] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.753311 21378 raft_consensus.cc:729] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.754096 21378 consensus_queue.cc:260] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.757380 21378 ts_tablet_manager.cc:1428] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260812 19:12:21.758496 21378 tablet_bootstrap.cc:492] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:21.760910 21379 tablet_bootstrap.cc:654] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.763937 21381 tablet_bootstrap.cc:654] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.765278 21378 tablet_bootstrap.cc:654] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.773947 21735 consensus_queue.cc:1035] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:21.778021 21378 tablet_bootstrap.cc:492] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:21.778585 21378 ts_tablet_manager.cc:1397] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20260812 19:12:21.789630 21378 raft_consensus.cc:348] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.790745 21378 raft_consensus.cc:374] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.791098 21378 raft_consensus.cc:729] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.791926 21378 consensus_queue.cc:260] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.795178 21378 ts_tablet_manager.cc:1428] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.016s user 0.010s sys 0.000s
I20260812 19:12:21.797626 21378 tablet_bootstrap.cc:492] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:21.809026 21735 consensus_queue.cc:1035] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:21.814597 21378 tablet_bootstrap.cc:654] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.816497 21734 consensus_queue.cc:1035] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:21.832367 21628 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:21.832872 21628 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.837263 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:21.838109 21237 raft_consensus.cc:2454] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:21.839133 21381 tablet_bootstrap.cc:492] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:21.839676 21381 ts_tablet_manager.cc:1397] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.101s user 0.019s sys 0.005s
I20260812 19:12:21.842967 21114 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:21.847023 21745 consensus_queue.cc:1035] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:21.847630 21712 raft_consensus.cc:2781] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:21.852864 21712 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:21.853585 21712 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:21.850464 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:21.848762 21628 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:21.857100 21378 tablet_bootstrap.cc:492] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:21.857774 21378 ts_tablet_manager.cc:1397] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.060s user 0.008s sys 0.007s
I20260812 19:12:21.858088 21570 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:21.858733 21570 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:21.859791 21112 leader_election.cc:341] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:21.860932 21378 raft_consensus.cc:348] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.861723 21378 raft_consensus.cc:374] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.862056 21378 raft_consensus.cc:729] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.861311 21570 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:21.863638 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:21.864933 21203 leader_election.cc:341] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:21.858440 21381 raft_consensus.cc:348] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.865733 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:21.867689 21381 raft_consensus.cc:374] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.871460 21381 raft_consensus.cc:729] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.873492 21379 tablet_bootstrap.cc:492] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:21.874207 21379 ts_tablet_manager.cc:1397] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.126s user 0.011s sys 0.007s
I20260812 19:12:21.871434 21378 consensus_queue.cc:260] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.876637 21378 ts_tablet_manager.cc:1428] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.019s user 0.003s sys 0.005s
I20260812 19:12:21.877774 21378 tablet_bootstrap.cc:492] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:21.873595 21381 consensus_queue.cc:260] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.877849 21712 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.880430 21381 ts_tablet_manager.cc:1428] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.040s user 0.005s sys 0.004s
I20260812 19:12:21.878880 21379 raft_consensus.cc:348] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.880906 21379 raft_consensus.cc:374] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.881206 21379 raft_consensus.cc:729] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.881624 21381 tablet_bootstrap.cc:492] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:21.883149 21379 consensus_queue.cc:260] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.885630 21379 ts_tablet_manager.cc:1428] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20260812 19:12:21.887033 21379 tablet_bootstrap.cc:492] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
W20260812 19:12:21.868755 21200 leader_election.cc:341] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:21.891654 21200 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:21.894038 21570 raft_consensus.cc:2726] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:21.894536 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:21.895270 21240 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:21.896004 21712 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:21.896874 21570 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:21.897508 21570 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.900640 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:21.915722 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:21.916630 21329 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
W20260812 19:12:21.917189 21203 leader_election.cc:341] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:21.918354 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:21.919309 21570 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:12:21.921747 21200 leader_election.cc:341] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:21.922253 21200 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:21.923022 21495 raft_consensus.cc:2726] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:21.923039 21240 raft_consensus.cc:2454] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:21.925477 21114 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:21.925895 21381 tablet_bootstrap.cc:654] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.926494 21628 raft_consensus.cc:2781] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:21.927094 21628 raft_consensus.cc:686] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:21.928155 21628 consensus_queue.cc:237] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.929370 21379 tablet_bootstrap.cc:654] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.929149 21329 raft_consensus.cc:2454] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:21.937037 21379 tablet_bootstrap.cc:492] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:21.943444 21379 ts_tablet_manager.cc:1397] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.012s
I20260812 19:12:21.945560 21378 tablet_bootstrap.cc:654] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.946458 21379 raft_consensus.cc:348] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.947312 21379 raft_consensus.cc:374] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.947569 21379 raft_consensus.cc:729] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.951654 21379 consensus_queue.cc:260] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.952229 21036 catalog_manager.cc:5495] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:21.958464 21419 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:21.959293 21419 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.961043 21381 tablet_bootstrap.cc:492] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:21.961548 21381 ts_tablet_manager.cc:1397] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.080s user 0.011s sys 0.007s
I20260812 19:12:21.964556 21381 raft_consensus.cc:348] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.965420 21381 raft_consensus.cc:374] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.965691 21381 raft_consensus.cc:729] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.966143 21378 tablet_bootstrap.cc:492] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:21.966353 21381 consensus_queue.cc:260] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.967408 21419 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:21.969784 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:21.973290 21381 ts_tablet_manager.cc:1428] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20260812 19:12:21.973450 21329 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:21.959314 21379 ts_tablet_manager.cc:1428] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.016s user 0.007s sys 0.001s
I20260812 19:12:21.966998 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:21.974809 21112 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:21.975929 21379 tablet_bootstrap.cc:492] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:21.976217 21239 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:21.975934 21712 raft_consensus.cc:2781] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:21.976976 21712 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:21.977339 21712 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:21.966652 21378 ts_tablet_manager.cc:1397] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.089s user 0.011s sys 0.003s
I20260812 19:12:21.980351 21381 tablet_bootstrap.cc:492] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:21.986585 21712 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.989813 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:21.990453 21240 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:21.992095 21378 raft_consensus.cc:348] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.992924 21378 raft_consensus.cc:374] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:21.993198 21378 raft_consensus.cc:729] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:21.993862 21378 consensus_queue.cc:260] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:21.998665 21379 tablet_bootstrap.cc:654] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:21.998769 21712 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:21.999053 21378 ts_tablet_manager.cc:1428] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.019s user 0.004s sys 0.003s
I20260812 19:12:21.999943 21378 tablet_bootstrap.cc:492] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:22.003540 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:22.004427 21329 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:22.005470 21381 tablet_bootstrap.cc:654] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
W20260812 19:12:22.007699 21287 consensus_peers.cc:487] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 -> Peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Couldn't send request to peer f741102bfe1441d1b453e5a838649059. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:22.008520 21240 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:22.009204 21378 tablet_bootstrap.cc:654] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:22.012205 21114 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:22.013463 21712 raft_consensus.cc:2781] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:22.014194 21712 raft_consensus.cc:686] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:22.015303 21712 consensus_queue.cc:237] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.023258 21379 tablet_bootstrap.cc:492] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:22.025048 21379 ts_tablet_manager.cc:1397] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.006s
I20260812 19:12:22.031554 21381 tablet_bootstrap.cc:492] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:22.031715 21329 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:22.032150 21381 ts_tablet_manager.cc:1397] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.004s
I20260812 19:12:22.032922 21378 tablet_bootstrap.cc:492] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:22.033474 21379 raft_consensus.cc:348] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.034248 21379 raft_consensus.cc:374] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:22.034513 21379 raft_consensus.cc:729] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:22.035195 21381 raft_consensus.cc:348] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.035212 21379 consensus_queue.cc:260] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.036027 21381 raft_consensus.cc:374] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:22.036314 21381 raft_consensus.cc:729] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:22.036955 21381 consensus_queue.cc:260] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.038816 21379 ts_tablet_manager.cc:1428] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.013s user 0.007s sys 0.002s
I20260812 19:12:22.033471 21378 ts_tablet_manager.cc:1397] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.011s
I20260812 19:12:22.040439 21381 ts_tablet_manager.cc:1428] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 19:12:22.041697 21379 tablet_bootstrap.cc:492] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:22.047286 21378 raft_consensus.cc:348] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.048066 21378 raft_consensus.cc:374] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:22.048362 21378 raft_consensus.cc:729] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:22.048985 21378 consensus_queue.cc:260] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.052381 21378 ts_tablet_manager.cc:1428] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20260812 19:12:22.051816 21037 catalog_manager.cc:5495] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:22.041728 21381 tablet_bootstrap.cc:492] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:22.054272 21378 tablet_bootstrap.cc:492] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:22.062960 21628 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:22.063557 21628 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.065981 21378 tablet_bootstrap.cc:654] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:22.075733 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:22.071128 21628 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:22.082484 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:22.083916 21328 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
W20260812 19:12:22.090745 21114 leader_election.cc:341] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260812 19:12:22.091609 21112 leader_election.cc:341] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:22.094013 21495 consensus_queue.cc:1035] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:12:22.096412 21734 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:22.097018 21734 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.098474 21495 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:22.092099 21112 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:22.101198 21379 tablet_bootstrap.cc:654] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:22.101250 21734 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:22.098973 21495 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.108500 21570 consensus_queue.cc:1035] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:22.111124 21570 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:22.111697 21570 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.112325 21495 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:22.114078 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:22.114863 21157 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:22.117316 21378 tablet_bootstrap.cc:492] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:22.117897 21378 ts_tablet_manager.cc:1397] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.064s user 0.001s sys 0.013s
I20260812 19:12:22.116807 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:22.101672 21628 raft_consensus.cc:2726] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:22.118728 21328 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:22.104586 21381 tablet_bootstrap.cc:654] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:22.117211 21203 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:22.121505 21747 raft_consensus.cc:2781] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:22.121038 21378 raft_consensus.cc:348] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.121981 21747 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:22.122056 21378 raft_consensus.cc:374] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:22.122411 21747 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:22.123270 21379 tablet_bootstrap.cc:492] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:22.123744 21379 ts_tablet_manager.cc:1397] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.082s user 0.007s sys 0.008s
I20260812 19:12:22.126237 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:22.127722 21157 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:22.128412 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:22.128983 21203 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:22.129225 21328 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:22.127134 21570 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:22.130867 21637 raft_consensus.cc:2781] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:22.139003 21637 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:22.139765 21637 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:22.131919 21747 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.122448 21378 raft_consensus.cc:729] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:22.147333 21378 consensus_queue.cc:260] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.148921 21747 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:22.149894 21378 ts_tablet_manager.cc:1428] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20260812 19:12:22.132361 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:22.150688 21240 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:22.156860 21378 tablet_bootstrap.cc:492] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:22.133411 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:22.133704 21735 consensus_queue.cc:1035] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:22.126720 21379 raft_consensus.cc:348] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.159103 21379 raft_consensus.cc:374] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:22.159416 21379 raft_consensus.cc:729] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
W20260812 19:12:22.160385 21287 leader_election.cc:341] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:22.160115 21379 consensus_queue.cc:260] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.162204 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:22.162492 21379 ts_tablet_manager.cc:1428] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.038s user 0.003s sys 0.002s
I20260812 19:12:22.162909 21156 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:22.163522 21379 tablet_bootstrap.cc:492] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:22.172845 21637 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.184649 21378 tablet_bootstrap.cc:654] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:22.193214 21379 tablet_bootstrap.cc:654] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:22.198391 21156 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:22.199870 21203 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:22.202371 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:22.195806 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:22.203161 21157 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:22.203399 21328 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:22.206525 21378 tablet_bootstrap.cc:492] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:22.206332 21287 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:22.206828 21797 raft_consensus.cc:2781] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:22.207155 21378 ts_tablet_manager.cc:1397] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.051s user 0.009s sys 0.004s
I20260812 19:12:22.207391 21797 raft_consensus.cc:686] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:22.208003 21735 raft_consensus.cc:2781] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:22.208593 21735 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:22.208364 21797 consensus_queue.cc:237] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.209072 21735 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:22.210479 21378 raft_consensus.cc:348] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.217048 21378 raft_consensus.cc:374] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:22.217356 21378 raft_consensus.cc:729] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:22.218050 21378 consensus_queue.cc:260] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.224355 21378 ts_tablet_manager.cc:1428] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260812 19:12:22.225490 21378 tablet_bootstrap.cc:492] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:22.227648 21570 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:22.228223 21570 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.238891 21796 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:22.239476 21796 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.242792 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:22.244314 21203 leader_election.cc:341] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:22.245077 21796 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:22.233502 21157 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:22.247803 21203 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:22.255789 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:22.253652 21795 raft_consensus.cc:2781] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:22.256871 21795 raft_consensus.cc:686] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:22.257863 21795 consensus_queue.cc:237] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.259307 21734 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:22.259840 21734 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.262087 21734 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:22.264245 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:22.264961 21157 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:22.266132 21287 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:12:22.264655 21203 leader_election.cc:341] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:22.268993 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:22.269758 21237 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:22.271658 21745 raft_consensus.cc:2781] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:12:22.272090 21745 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:22.272445 21745 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:22.274569 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:22.275336 21319 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:22.277053 21200 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:22.288211 21637 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:22.288796 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:22.284969 21792 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:22.285236 21570 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:12:22.293203 21200 leader_election.cc:341] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:22.293696 21200 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:22.297139 21381 tablet_bootstrap.cc:492] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:22.297700 21381 ts_tablet_manager.cc:1397] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.256s user 0.032s sys 0.069s
I20260812 19:12:22.290442 21792 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:22.299916 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:22.300601 21327 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:22.301060 21745 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.301591 21381 raft_consensus.cc:348] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.302117 21381 raft_consensus.cc:374] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:22.302285 21381 raft_consensus.cc:729] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:22.302868 21381 consensus_queue.cc:260] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.303313 21792 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:22.304630 21381 ts_tablet_manager.cc:1428] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.007s user 0.000s sys 0.004s
I20260812 19:12:22.282598 21328 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:22.315093 21381 tablet_bootstrap.cc:492] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:22.317651 21327 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:22.322942 21378 tablet_bootstrap.cc:654] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:22.304678 21570 raft_consensus.cc:2726] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:22.325132 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:22.325735 21157 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:22.291232 21735 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.324182 21037 catalog_manager.cc:5495] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:22.332886 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:12:22.338330 21287 leader_election.cc:341] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:22.364706 21157 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:22.365437 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:22.366276 21239 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:22.366675 21287 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:22.367523 21494 raft_consensus.cc:2781] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:22.373680 21381 tablet_bootstrap.cc:654] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:22.363821 21745 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:22.382512 21037 catalog_manager.cc:5495] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:22.404757 21735 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:22.417968 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:22.422525 21236 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:22.438469 21239 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:22.439968 21628 consensus_queue.cc:1035] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:22.412823 21494 raft_consensus.cc:686] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:22.448073 21378 tablet_bootstrap.cc:492] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:22.448642 21378 ts_tablet_manager.cc:1397] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.223s user 0.016s sys 0.001s
I20260812 19:12:22.447027 21494 consensus_queue.cc:237] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.452035 21378 raft_consensus.cc:348] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.448096 21379 tablet_bootstrap.cc:492] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:22.452814 21378 raft_consensus.cc:374] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:22.453125 21378 raft_consensus.cc:729] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:22.453161 21379 ts_tablet_manager.cc:1397] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.290s user 0.014s sys 0.017s
I20260812 19:12:22.453822 21378 consensus_queue.cc:260] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.456593 21236 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:22.456869 21379 raft_consensus.cc:348] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.457660 21379 raft_consensus.cc:374] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:22.457980 21379 raft_consensus.cc:729] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:22.458397 21570 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:22.458746 21379 consensus_queue.cc:260] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.458932 21570 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.460103 21287 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:22.461205 21628 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:22.462661 21379 ts_tablet_manager.cc:1428] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20260812 19:12:22.463186 21570 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:22.463915 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:22.464891 21804 raft_consensus.cc:2781] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:22.465451 21804 raft_consensus.cc:686] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
W20260812 19:12:22.465507 21203 leader_election.cc:341] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:22.465938 21378 ts_tablet_manager.cc:1428] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.017s user 0.007s sys 0.002s
I20260812 19:12:22.461879 21628 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.468374 21379 tablet_bootstrap.cc:492] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:22.473066 21378 tablet_bootstrap.cc:492] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:22.466481 21804 consensus_queue.cc:237] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.468854 21792 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.484069 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:22.484318 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:22.484974 21328 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:22.485481 21236 raft_consensus.cc:2379] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 1: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:12:22.487628 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:22.489754 21792 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:22.496845 21628 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:22.493512 21712 consensus_queue.cc:1035] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260812 19:12:22.509878 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:22.510514 21157 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:22.513307 21112 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:22.515013 21381 tablet_bootstrap.cc:492] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:22.515610 21381 ts_tablet_manager.cc:1397] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.201s user 0.023s sys 0.027s
I20260812 19:12:22.519901 21157 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:22.521420 21712 raft_consensus.cc:2741] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260812 19:12:22.526857 21203 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:22.529460 21795 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:22.530824 21795 raft_consensus.cc:686] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:12:22.536485 21200 leader_election.cc:341] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:22.537201 21200 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:22.541735 21378 tablet_bootstrap.cc:654] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:22.546283 21381 raft_consensus.cc:348] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.547367 21381 raft_consensus.cc:374] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:22.547704 21381 raft_consensus.cc:729] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:22.548401 21381 consensus_queue.cc:260] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.539283 21747 raft_consensus.cc:2726] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:22.551836 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:22.552592 21327 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:22.554867 21795 consensus_queue.cc:237] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.556713 21712 consensus_queue.cc:1035] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:12:22.558372 21037 catalog_manager.cc:5495] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 2, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 2 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:22.566097 21381 ts_tablet_manager.cc:1428] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.050s user 0.006s sys 0.002s
I20260812 19:12:22.593675 21381 tablet_bootstrap.cc:492] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:22.594789 21734 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:22.599910 21734 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.609928 21734 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:22.611248 21747 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:22.614194 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:22.614996 21157 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:22.617147 21287 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:22.622357 21795 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:22.623003 21795 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.625056 21795 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:22.612299 21628 consensus_queue.cc:1035] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260812 19:12:22.627236 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:22.628028 21239 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:22.611827 21747 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.630836 21747 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:22.634430 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:22.638195 21379 tablet_bootstrap.cc:654] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
W20260812 19:12:22.642395 21200 leader_election.cc:341] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:22.647171 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:22.647994 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:22.648583 21329 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:12:22.660023 21200 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:22.642189 21734 raft_consensus.cc:2781] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:22.664412 21570 raft_consensus.cc:2781] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:12:22.668399 21570 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:22.663875 21734 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:22.659791 21327 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:22.673146 21570 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:12:22.678785 21734 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:22.702121 21381 tablet_bootstrap.cc:654] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:22.703653 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:22.793848 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 56794.3us
I20260812 19:12:22.727690 21792 consensus_queue.cc:1035] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20260812 19:12:22.795477 21203 leader_election.cc:341] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:22.798300 21203 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:22.807090 21495 raft_consensus.cc:2726] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:22.871023 21378 tablet_bootstrap.cc:492] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:22.871660 21378 ts_tablet_manager.cc:1397] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.403s user 0.022s sys 0.019s
I20260812 19:12:22.886685 21734 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.889637 21734 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:22.913064 21379 tablet_bootstrap.cc:492] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:22.913707 21379 ts_tablet_manager.cc:1397] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.446s user 0.036s sys 0.021s
W20260812 19:12:22.934720 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 107524us
I20260812 19:12:22.942871 21378 raft_consensus.cc:348] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.944245 21378 raft_consensus.cc:374] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:22.944653 21378 raft_consensus.cc:729] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:22.945658 21378 consensus_queue.cc:260] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:22.910099 21570 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:22.981873 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 162805us
I20260812 19:12:22.993247 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:22.994161 21157 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:22.981334 21712 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:12:23.001463 21712 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.052146 21157 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:23.056682 21378 ts_tablet_manager.cc:1428] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.185s user 0.011s sys 0.000s
I20260812 19:12:23.058375 21378 tablet_bootstrap.cc:492] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:23.000532 21792 consensus_queue.cc:1035] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.059s
I20260812 19:12:23.005952 21747 consensus_queue.cc:1035] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.057s
I20260812 19:12:23.068215 21379 raft_consensus.cc:348] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.069221 21379 raft_consensus.cc:374] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:23.069533 21379 raft_consensus.cc:729] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:23.070211 21379 consensus_queue.cc:260] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:23.075212 21822 log.cc:924] Time spent T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
W20260812 19:12:23.080931 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 98045.9us
I20260812 19:12:23.059056 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:23.094944 21379 ts_tablet_manager.cc:1428] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.181s user 0.009s sys 0.000s
I20260812 19:12:23.089300 21237 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:23.105545 21381 tablet_bootstrap.cc:492] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:23.089099 21712 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:23.118700 21287 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:23.138441 21837 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:23.139298 21381 ts_tablet_manager.cc:1397] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.551s user 0.018s sys 0.020s
I20260812 19:12:23.106809 21811 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:23.142589 21379 tablet_bootstrap.cc:492] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412: Bootstrap starting.
W20260812 19:12:23.144502 21287 consensus_peers.cc:487] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 -> Peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Couldn't send request to peer f741102bfe1441d1b453e5a838649059. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:23.159504 21804 raft_consensus.cc:2781] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:23.145663 21811 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.186582 21804 raft_consensus.cc:686] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:23.167688 21155 raft_consensus.cc:1261] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260812 19:12:23.187871 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:23.192919 21329 raft_consensus.cc:2379] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:12:23.217142 21839 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:23.221876 21811 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:23.179354 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:23.218006 21839 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.265270 21570 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:23.273761 21239 raft_consensus.cc:2354] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4.
I20260812 19:12:23.276986 21381 raft_consensus.cc:348] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.280583 21381 raft_consensus.cc:374] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:23.278149 21835 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:23.281576 21381 raft_consensus.cc:729] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:23.282467 21381 consensus_queue.cc:260] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.288940 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:23.289893 21157 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 4
I20260812 19:12:23.281847 21835 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.296645 21237 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:23.292608 21804 consensus_queue.cc:237] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.306133 21157 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:12:23.286893 21735 consensus_queue.cc:1035] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260812 19:12:23.307964 21203 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:23.301405 21378 tablet_bootstrap.cc:654] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:23.311283 21833 raft_consensus.cc:2781] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:12:23.311889 21833 raft_consensus.cc:686] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:23.312788 21833 consensus_queue.cc:237] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.321170 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:23.321993 21238 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:23.325958 21379 tablet_bootstrap.cc:654] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:23.329020 21381 ts_tablet_manager.cc:1428] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.189s user 0.015s sys 0.001s
I20260812 19:12:23.330127 21287 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:23.332476 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:23.317200 21839 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:23.333304 21836 raft_consensus.cc:2781] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:23.339885 21836 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:23.340502 21836 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:23.349798 21835 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:23.406040 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:23.406853 21239 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:23.407109 21570 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:12:23.407430 21287 leader_election.cc:341] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:23.407704 21570 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.411568 21287 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:23.413118 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:23.414346 21157 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:23.421335 21834 raft_consensus.cc:2781] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:23.421882 21834 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:23.422339 21834 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:23.431629 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:23.439030 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:23.444301 21833 consensus_queue.cc:1035] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:23.445165 21796 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:12:23.438110 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:23.453047 21319 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:12:23.424211 21570 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:23.447160 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:23.462963 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:23.463819 21236 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
W20260812 19:12:23.471640 21093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260812 19:12:23.487387 21379 tablet_bootstrap.cc:492] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:23.487946 21379 ts_tablet_manager.cc:1397] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.374s user 0.021s sys 0.021s
I20260812 19:12:23.497133 21379 raft_consensus.cc:348] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.498102 21379 raft_consensus.cc:374] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:23.501113 21114 leader_election.cc:398] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4.
I20260812 19:12:23.501507 21114 leader_election.cc:401] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:12:23.498420 21379 raft_consensus.cc:729] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:23.506455 21379 consensus_queue.cc:260] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:23.509248 21203 leader_election.cc:341] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:23.509881 21114 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:23.477867 21836 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.520596 21840 raft_consensus.cc:2781] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:23.521291 21840 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:23.521937 21378 tablet_bootstrap.cc:492] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:23.522555 21381 tablet_bootstrap.cc:492] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:23.522565 21378 ts_tablet_manager.cc:1397] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.465s user 0.007s sys 0.018s
W20260812 19:12:23.531782 21200 leader_election.cc:341] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:23.538844 21200 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:23.541016 21495 raft_consensus.cc:2726] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:23.541817 21834 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:23.549983 21846 log.cc:924] Time spent T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20260812 19:12:23.546314 21820 raft_consensus.cc:2726] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4.
I20260812 19:12:23.522030 21840 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:23.557482 21379 ts_tablet_manager.cc:1428] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.069s user 0.008s sys 0.002s
I20260812 19:12:23.551718 21378 raft_consensus.cc:348] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.562635 21378 raft_consensus.cc:374] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:23.562987 21378 raft_consensus.cc:729] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:23.560527 21035 catalog_manager.cc:5495] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:23.564013 21378 consensus_queue.cc:260] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.583472 21243 raft_consensus.cc:1261] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260812 19:12:23.572333 21328 raft_consensus.cc:2379] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 1: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:12:23.585469 21378 ts_tablet_manager.cc:1428] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.061s user 0.008s sys 0.000s
I20260812 19:12:23.609186 21828 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:23.612011 21379 tablet_bootstrap.cc:492] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:23.616614 21838 consensus_queue.cc:1035] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20260812 19:12:23.619590 21319 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:12:23.621666 21378 tablet_bootstrap.cc:492] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:23.610391 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:23.612689 21834 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:23.609930 21828 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.712221 21836 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:23.721709 21834 consensus_queue.cc:1035] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:23.835323 21828 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:12:23.849480 21855 log.cc:924] Time spent T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20260812 19:12:23.865064 21379 tablet_bootstrap.cc:654] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:23.874272 21381 tablet_bootstrap.cc:654] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:23.888101 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:23.888980 21244 raft_consensus.cc:2379] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 1: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:12:23.935058 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:23.941087 21840 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.951869 21156 raft_consensus.cc:2379] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 1: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:12:23.955889 21685 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:23.956738 21685 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.959419 21154 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:23.962088 21827 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:23.948153 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:23.965721 21378 tablet_bootstrap.cc:654] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:23.966310 21840 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:23.967983 21241 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:23.973454 21685 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:23.974943 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:23.975831 21244 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:23.962687 21827 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:23.997767 21734 consensus_queue.cc:1035] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260812 19:12:24.016819 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:24.012621 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:24.020697 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:24.028047 21319 raft_consensus.cc:2379] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 1: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:12:24.016923 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:24.030830 21240 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:24.020886 21241 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:24.039387 21112 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:24.040144 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:24.036782 21244 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
W20260812 19:12:24.048429 21287 leader_election.cc:341] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:24.051234 21287 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:24.053515 21734 raft_consensus.cc:2726] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260812 19:12:24.056453 21114 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:12:24.058245 21381 tablet_bootstrap.cc:492] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:24.058878 21381 ts_tablet_manager.cc:1397] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.537s user 0.023s sys 0.019s
I20260812 19:12:24.060105 21827 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:24.065735 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:24.067267 21836 consensus_queue.cc:1035] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260812 19:12:24.080411 21322 raft_consensus.cc:1261] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260812 19:12:24.062288 21819 raft_consensus.cc:2781] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:24.093513 21819 raft_consensus.cc:686] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:24.094826 21819 consensus_queue.cc:237] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:24.097364 21158 raft_consensus.cc:1261] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260812 19:12:24.081864 21203 leader_election.cc:341] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
W20260812 19:12:24.101104 21287 leader_election.cc:341] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:24.111271 21287 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:24.112936 21287 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:24.114078 21034 catalog_manager.cc:5495] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:24.124727 21685 raft_consensus.cc:2781] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:24.125422 21685 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:24.125885 21685 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:24.130784 21834 raft_consensus.cc:2781] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:24.133338 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:24.137030 21834 raft_consensus.cc:686] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:24.138145 21834 consensus_queue.cc:237] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:24.109881 21792 consensus_queue.cc:1035] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.034s
I20260812 19:12:24.169481 21381 raft_consensus.cc:348] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:24.170639 21381 raft_consensus.cc:374] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:24.171042 21381 raft_consensus.cc:729] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:24.172120 21381 consensus_queue.cc:260] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:24.174541 21381 ts_tablet_manager.cc:1428] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.115s user 0.008s sys 0.000s
I20260812 19:12:24.176805 21381 tablet_bootstrap.cc:492] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:24.215631 21381 tablet_bootstrap.cc:654] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
W20260812 19:12:24.236792 21874 log.cc:924] Time spent T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.064s user 0.004s sys 0.000s
I20260812 19:12:24.225726 21685 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:24.268253 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 106935us
I20260812 19:12:24.271171 21329 raft_consensus.cc:2379] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 1: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
W20260812 19:12:24.280563 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 147530us
I20260812 19:12:24.284282 21379 tablet_bootstrap.cc:492] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:24.285111 21379 ts_tablet_manager.cc:1397] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.693s user 0.021s sys 0.016s
I20260812 19:12:24.264206 21827 consensus_queue.cc:1035] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 0, Time since last communication: 0.023s
I20260812 19:12:24.304726 21811 raft_consensus.cc:2726] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:24.307274 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:24.308516 21378 tablet_bootstrap.cc:492] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:24.309058 21378 ts_tablet_manager.cc:1397] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.704s user 0.014s sys 0.023s
W20260812 19:12:24.310199 21287 leader_election.cc:341] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:24.345901 21378 raft_consensus.cc:348] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:24.346922 21378 raft_consensus.cc:374] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:24.347234 21378 raft_consensus.cc:729] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:24.348027 21378 consensus_queue.cc:260] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:24.354637 21379 raft_consensus.cc:348] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:24.355708 21379 raft_consensus.cc:374] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:24.356029 21379 raft_consensus.cc:729] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:24.368374 21379 consensus_queue.cc:260] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:24.367342 21685 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:24.381415 21378 ts_tablet_manager.cc:1428] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.072s user 0.004s sys 0.004s
I20260812 19:12:24.388012 21034 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48434 (request call id 93) took 1438 ms. Trace:
W20260812 19:12:24.389879 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 108161us
I20260812 19:12:24.395792 21378 tablet_bootstrap.cc:492] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:24.395641 21818 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:24.398406 21818 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:24.401939 21379 ts_tablet_manager.cc:1428] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.116s user 0.008s sys 0.000s
I20260812 19:12:24.406302 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:24.407016 21244 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:24.421789 21379 tablet_bootstrap.cc:492] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:24.425981 21200 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:24.459844 21244 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
W20260812 19:12:24.482492 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 48532.2us
I20260812 19:12:24.389210 21034 rpcz_store.cc:276] 0812 19:12:22.948912 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:22.969654 (+ 20742us) service_pool.cc:227] Handling call
0812 19:12:24.386534 (+1416880us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0812 19:12:24.156780 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0812 19:12:24.157725 (+ 945us) write_op.cc:432] Acquiring schema lock in shared mode
0812 19:12:24.157761 (+ 36us) write_op.cc:435] Acquired schema lock
0812 19:12:24.157779 (+ 18us) tablet.cc:660] Decoding operations
0812 19:12:24.158183 (+ 404us) write_op.cc:620] Acquiring the partition lock for write op
0812 19:12:24.158254 (+ 71us) write_op.cc:641] Partition lock acquired for write op
0812 19:12:24.158279 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0812 19:12:24.158514 (+ 235us) tablet.cc:699] Row locks acquired
0812 19:12:24.158534 (+ 20us) write_op.cc:260] PREPARE: finished
0812 19:12:24.158706 (+ 172us) write_op.cc:270] Start()
0812 19:12:24.158838 (+ 132us) write_op.cc:276] Timestamp: P: 1786561944158661 usec, L: 0
0812 19:12:24.158854 (+ 16us) op_driver.cc:331] REPLICATION: starting
0812 19:12:24.159457 (+ 603us) log.cc:842] Serialized 540 byte log entry
0812 19:12:24.191960 (+ 32503us) op_driver.cc:444] REPLICATION: finished
0812 19:12:24.212576 (+ 20616us) write_op.cc:301] APPLY: starting
0812 19:12:24.215451 (+ 2875us) tablet.cc:1365] starting BulkCheckPresence
0812 19:12:24.220553 (+ 5102us) tablet.cc:1368] finished BulkCheckPresence
0812 19:12:24.220743 (+ 190us) tablet.cc:1370] starting ApplyRowOperation cycle
0812 19:12:24.221006 (+ 263us) tablet.cc:1381] finished ApplyRowOperation cycle
0812 19:12:24.221240 (+ 234us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0812 19:12:24.221311 (+ 71us) write_op.cc:312] APPLY: finished
0812 19:12:24.221583 (+ 272us) log.cc:842] Serialized 22 byte log entry
0812 19:12:24.221817 (+ 234us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:12:24.221934 (+ 117us) write_op.cc:454] Released schema lock
0812 19:12:24.221982 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0812 19:12:24.222335 (+ 353us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:12:24.222375 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":381066,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":8238,"apply.run_cpu_time_us":2483,"apply.run_wall_time_us":11260,"num_ops":1,"prepare.queue_time_us":365,"prepare.run_cpu_time_us":2990,"prepare.run_wall_time_us":2991,"replication_time_us":29933}]]}
I20260812 19:12:24.470988 21829 raft_consensus.cc:2726] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:24.498936 21287 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:24.512744 21818 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:24.528807 21841 raft_consensus.cc:2781] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:24.530992 21841 raft_consensus.cc:686] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:24.532158 21841 consensus_queue.cc:237] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:24.612143 21287 consensus_peers.cc:487] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 -> Peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Couldn't send request to peer f741102bfe1441d1b453e5a838649059. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:24.644367 21734 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:24.676513 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:24.673599 21734 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:24.749385 21378 tablet_bootstrap.cc:654] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:24.769132 21379 tablet_bootstrap.cc:654] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:24.781275 21734 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:24.782382 21381 tablet_bootstrap.cc:492] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:24.785704 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:24.789840 21834 consensus_queue.cc:1035] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:24.794306 21381 ts_tablet_manager.cc:1397] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.618s user 0.031s sys 0.046s
W20260812 19:12:24.802114 21093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260812 19:12:24.803251 21287 leader_election.cc:341] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:24.817258 21381 raft_consensus.cc:348] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:24.820231 21381 raft_consensus.cc:374] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:24.820827 21381 raft_consensus.cc:729] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:24.822562 21381 consensus_queue.cc:260] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:24.824941 21898 log.cc:924] Time spent T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20260812 19:12:24.804620 21837 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:24.836755 21158 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:24.845170 21795 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:24.855086 21795 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:24.857741 21795 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:24.848680 21837 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:24.861814 21837 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:24.872452 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:24.873251 21381 ts_tablet_manager.cc:1428] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.078s user 0.008s sys 0.003s
I20260812 19:12:24.873781 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:24.875734 21203 leader_election.cc:341] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:24.877252 21155 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:24.882236 21829 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:24.883092 21829 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:24.885555 21829 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:24.887261 21203 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:24.888684 21747 raft_consensus.cc:2781] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:24.889354 21747 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:24.889926 21747 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:24.891738 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:24.908811 21820 consensus_queue.cc:1035] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:12:24.909277 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:24.909564 21203 leader_election.cc:341] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:24.912837 21379 tablet_bootstrap.cc:492] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:24.913358 21379 ts_tablet_manager.cc:1397] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.500s user 0.023s sys 0.048s
W20260812 19:12:24.920291 21287 leader_election.cc:341] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:24.928404 21287 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:24.943543 21841 raft_consensus.cc:2726] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:24.967320 21378 tablet_bootstrap.cc:492] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:24.967905 21378 ts_tablet_manager.cc:1397] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.582s user 0.036s sys 0.041s
I20260812 19:12:24.971587 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:24.972997 21378 raft_consensus.cc:348] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:24.973790 21378 raft_consensus.cc:374] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:24.974076 21378 raft_consensus.cc:729] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:24.974893 21378 consensus_queue.cc:260] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:24.977171 21379 raft_consensus.cc:348] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:24.977988 21379 raft_consensus.cc:374] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:24.972602 21381 tablet_bootstrap.cc:492] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:24.978356 21379 raft_consensus.cc:729] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:24.986989 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:24.978591 21034 catalog_manager.cc:5495] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:24.983687 21378 ts_tablet_manager.cc:1428] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260812 19:12:24.985625 21379 consensus_queue.cc:260] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.002653 21747 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.006161 21381 tablet_bootstrap.cc:654] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:25.007476 21747 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:25.007783 21321 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:24.997997 21034 catalog_manager.cc:5495] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:25.009956 21379 ts_tablet_manager.cc:1428] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.096s user 0.009s sys 0.004s
I20260812 19:12:25.011026 21379 tablet_bootstrap.cc:492] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:25.013228 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:25.014456 21157 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:25.015528 21378 tablet_bootstrap.cc:492] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:25.024771 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:25.036309 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:12:25.041890 21287 consensus_peers.cc:487] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 -> Peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Couldn't send request to peer f741102bfe1441d1b453e5a838649059. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:25.043133 21157 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:25.044276 21203 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:25.053568 21891 raft_consensus.cc:2781] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:25.054193 21891 raft_consensus.cc:686] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:25.062605 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:25.063293 21329 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:25.077080 21381 tablet_bootstrap.cc:492] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:25.077602 21381 ts_tablet_manager.cc:1397] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.105s user 0.017s sys 0.013s
I20260812 19:12:25.055224 21891 consensus_queue.cc:237] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.133797 21329 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:25.139833 21381 raft_consensus.cc:348] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.147890 21381 raft_consensus.cc:374] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:25.148231 21381 raft_consensus.cc:729] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:25.148900 21381 consensus_queue.cc:260] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.153954 21114 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:25.154331 21381 ts_tablet_manager.cc:1428] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.076s user 0.009s sys 0.001s
I20260812 19:12:25.156212 21892 raft_consensus.cc:2726] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:25.175195 21840 consensus_queue.cc:1035] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
W20260812 19:12:25.213557 21200 leader_election.cc:341] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:25.219235 21200 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:25.220022 21381 tablet_bootstrap.cc:492] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
W20260812 19:12:25.224857 21200 leader_election.cc:341] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:25.225517 21379 tablet_bootstrap.cc:654] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:25.225375 21200 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:25.226482 21837 raft_consensus.cc:2726] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:25.228550 21797 raft_consensus.cc:2726] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:25.231179 21378 tablet_bootstrap.cc:654] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:25.246847 21902 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:25.247458 21902 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.254192 21902 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:25.279816 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:25.291883 21203 leader_election.cc:341] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:25.357053 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:12:25.361225 21200 leader_election.cc:341] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:25.361867 21200 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:25.368790 21796 raft_consensus.cc:2726] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:25.372323 21834 consensus_queue.cc:1035] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260812 19:12:25.394861 21570 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:25.406184 21570 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.408412 21570 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:25.409608 21381 tablet_bootstrap.cc:654] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:25.417650 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:25.420154 21379 tablet_bootstrap.cc:492] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:25.420881 21379 ts_tablet_manager.cc:1397] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.410s user 0.011s sys 0.011s
I20260812 19:12:25.428306 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:25.429008 21379 raft_consensus.cc:348] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.429755 21379 raft_consensus.cc:374] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260812 19:12:25.431133 21200 leader_election.cc:341] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:25.432862 21379 raft_consensus.cc:729] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
W20260812 19:12:25.438969 21203 leader_election.cc:341] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:25.439477 21203 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:25.440557 21378 tablet_bootstrap.cc:492] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:25.441072 21378 ts_tablet_manager.cc:1397] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.442s user 0.021s sys 0.012s
I20260812 19:12:25.441763 21495 raft_consensus.cc:2726] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:25.444621 21150 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:25.442716 21379 consensus_queue.cc:260] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.452303 21378 raft_consensus.cc:348] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.456908 21378 raft_consensus.cc:374] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:25.457190 21378 raft_consensus.cc:729] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:25.457723 21378 consensus_queue.cc:260] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.462256 21378 ts_tablet_manager.cc:1428] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.021s user 0.005s sys 0.002s
I20260812 19:12:25.463358 21378 tablet_bootstrap.cc:492] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:25.478693 21379 ts_tablet_manager.cc:1428] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.057s user 0.008s sys 0.002s
I20260812 19:12:25.480129 21379 tablet_bootstrap.cc:492] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:25.494004 21381 tablet_bootstrap.cc:492] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:25.494529 21381 ts_tablet_manager.cc:1397] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.282s user 0.006s sys 0.017s
I20260812 19:12:25.496826 21818 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:12:25.498283 21818 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.498553 21381 raft_consensus.cc:348] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.499405 21381 raft_consensus.cc:374] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:25.499719 21381 raft_consensus.cc:729] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:25.500609 21381 consensus_queue.cc:260] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.503293 21381 ts_tablet_manager.cc:1428] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260812 19:12:25.504532 21381 tablet_bootstrap.cc:492] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:25.510670 21379 tablet_bootstrap.cc:654] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:25.511678 21378 tablet_bootstrap.cc:654] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:25.512127 21381 tablet_bootstrap.cc:654] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:25.524693 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:25.525687 21818 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:25.527228 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:25.533028 21037 catalog_manager.cc:5495] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:25.533030 21112 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:25.544157 21379 tablet_bootstrap.cc:492] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:25.544716 21379 ts_tablet_manager.cc:1397] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.065s user 0.012s sys 0.000s
I20260812 19:12:25.554728 21035 catalog_manager.cc:5495] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:25.561672 21818 raft_consensus.cc:2726] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:12:25.568495 21379 raft_consensus.cc:348] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.569435 21379 raft_consensus.cc:374] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:25.569720 21379 raft_consensus.cc:729] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:25.569757 21805 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:25.570314 21805 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.570411 21379 consensus_queue.cc:260] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.572365 21805 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:25.573004 21379 ts_tablet_manager.cc:1428] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.028s user 0.006s sys 0.001s
I20260812 19:12:25.574128 21379 tablet_bootstrap.cc:492] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:25.579072 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:25.579998 21241 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:25.578596 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:25.583602 21287 leader_election.cc:341] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:25.584317 21287 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:25.585410 21685 raft_consensus.cc:2781] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:25.585819 21685 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:25.586166 21685 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:25.600472 21839 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:25.603060 21495 consensus_queue.cc:1035] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:12:25.603561 21379 tablet_bootstrap.cc:654] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:25.601048 21839 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.609865 21839 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:25.610972 21826 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:25.611541 21826 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.614682 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:25.615206 21826 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:25.615787 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:12:25.617081 21203 leader_election.cc:341] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:25.617627 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:12:25.618412 21287 leader_election.cc:341] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:25.618595 21322 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 0.
I20260812 19:12:25.619375 21037 catalog_manager.cc:5495] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } }
I20260812 19:12:25.620311 21200 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:25.622460 21495 raft_consensus.cc:2781] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:25.622961 21495 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:25.623301 21495 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:25.622052 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:25.625144 21241 raft_consensus.cc:2454] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:25.636965 21378 tablet_bootstrap.cc:492] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:25.637506 21378 ts_tablet_manager.cc:1397] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.174s user 0.008s sys 0.027s
I20260812 19:12:25.633824 21685 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.649562 21287 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:25.651754 21495 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.659947 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:12:25.662035 21203 leader_election.cc:341] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:25.661211 21378 raft_consensus.cc:348] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.662585 21378 raft_consensus.cc:374] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:25.662957 21378 raft_consensus.cc:729] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:25.664857 21378 consensus_queue.cc:260] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.653868 21495 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:25.677315 21381 tablet_bootstrap.cc:492] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:25.677870 21381 ts_tablet_manager.cc:1397] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.174s user 0.019s sys 0.019s
I20260812 19:12:25.682309 21381 raft_consensus.cc:348] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.683344 21381 raft_consensus.cc:374] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:25.683677 21381 raft_consensus.cc:729] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:25.683573 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:25.684432 21320 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:25.684592 21381 consensus_queue.cc:260] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.679100 21378 ts_tablet_manager.cc:1428] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.041s user 0.009s sys 0.001s
I20260812 19:12:25.688542 21378 tablet_bootstrap.cc:492] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:25.692495 21685 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:25.675143 21839 raft_consensus.cc:2781] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:25.695289 21839 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:25.697062 21839 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:25.700549 21792 consensus_queue.cc:1035] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:25.702498 21381 ts_tablet_manager.cc:1428] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.024s user 0.006s sys 0.002s
I20260812 19:12:25.705466 21381 tablet_bootstrap.cc:492] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:25.716116 21151 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:25.724110 21037 catalog_manager.cc:5495] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } }
I20260812 19:12:25.728879 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:25.735327 21378 tablet_bootstrap.cc:654] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
W20260812 19:12:25.736518 21287 leader_election.cc:341] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:25.731038 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:25.754424 21381 tablet_bootstrap.cc:654] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:25.762260 21379 tablet_bootstrap.cc:492] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:25.762874 21379 ts_tablet_manager.cc:1397] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.189s user 0.017s sys 0.004s
I20260812 19:12:25.742532 21320 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:12:25.768440 21200 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:25.769683 21891 raft_consensus.cc:2781] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:25.769416 21379 raft_consensus.cc:348] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.770236 21891 raft_consensus.cc:686] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:25.771353 21891 consensus_queue.cc:237] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.770352 21379 raft_consensus.cc:374] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:25.778594 21379 raft_consensus.cc:729] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:25.779415 21379 consensus_queue.cc:260] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.760476 21241 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:25.793546 21379 ts_tablet_manager.cc:1428] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20260812 19:12:25.794661 21379 tablet_bootstrap.cc:492] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:25.800084 21839 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.830524 21378 tablet_bootstrap.cc:492] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:25.831086 21378 ts_tablet_manager.cc:1397] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.143s user 0.017s sys 0.013s
I20260812 19:12:25.833284 21037 catalog_manager.cc:5495] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:25.839648 21241 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:25.845849 21819 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:25.846436 21819 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.851991 21839 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:25.853999 21819 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:25.855134 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:25.858093 21320 raft_consensus.cc:2379] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 1: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:12:25.860042 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:25.861256 21378 raft_consensus.cc:348] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.862221 21287 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:25.868029 21378 raft_consensus.cc:374] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:25.868395 21378 raft_consensus.cc:729] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:25.871852 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:25.872750 21243 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:25.875834 21381 tablet_bootstrap.cc:492] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:25.876323 21381 ts_tablet_manager.cc:1397] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.173s user 0.019s sys 0.002s
I20260812 19:12:25.873775 21839 raft_consensus.cc:2781] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:25.875214 21378 consensus_queue.cc:260] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.861021 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:25.897056 21378 ts_tablet_manager.cc:1428] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.066s user 0.006s sys 0.002s
I20260812 19:12:25.896829 21114 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:25.883565 21839 raft_consensus.cc:686] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
W20260812 19:12:25.901252 21287 leader_election.cc:341] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:25.906250 21379 tablet_bootstrap.cc:654] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:25.907526 21883 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:25.911820 21839 consensus_queue.cc:237] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.919809 21378 tablet_bootstrap.cc:492] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa: Bootstrap starting.
I20260812 19:12:25.928740 21381 raft_consensus.cc:348] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.938170 21381 raft_consensus.cc:374] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:25.938498 21381 raft_consensus.cc:729] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
W20260812 19:12:25.941198 21093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260812 19:12:25.943331 21381 consensus_queue.cc:260] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:25.938280 21883 raft_consensus.cc:2726] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:25.960860 21381 ts_tablet_manager.cc:1428] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.084s user 0.010s sys 0.000s
W20260812 19:12:25.972956 21944 log.cc:924] Time spent T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260812 19:12:25.974059 21243 raft_consensus.cc:2454] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:25.979703 21379 tablet_bootstrap.cc:492] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:25.980741 21379 ts_tablet_manager.cc:1397] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.186s user 0.012s sys 0.017s
I20260812 19:12:25.988551 21381 tablet_bootstrap.cc:492] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:25.999066 21287 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:26.004181 21685 raft_consensus.cc:2781] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:26.003940 21379 raft_consensus.cc:348] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.004866 21379 raft_consensus.cc:374] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:26.005179 21379 raft_consensus.cc:729] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:26.005918 21379 consensus_queue.cc:260] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.008507 21826 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:26.009605 21826 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.010037 21379 ts_tablet_manager.cc:1428] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.029s user 0.006s sys 0.001s
I20260812 19:12:26.011191 21379 tablet_bootstrap.cc:492] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:26.011528 21037 catalog_manager.cc:5495] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:26.012758 21685 raft_consensus.cc:686] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:26.013813 21685 consensus_queue.cc:237] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.075304 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:26.076442 21826 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:26.075810 21378 tablet_bootstrap.cc:654] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa: Neither blocks nor log segments found. Creating new log.
W20260812 19:12:26.086143 21203 leader_election.cc:341] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:26.090636 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:12:26.097530 21200 leader_election.cc:341] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:26.098053 21200 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:26.099315 21891 raft_consensus.cc:2726] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:26.104218 21379 tablet_bootstrap.cc:654] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
W20260812 19:12:26.104755 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 90852.1us
I20260812 19:12:26.106997 21378 tablet_bootstrap.cc:492] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa: No bootstrap required, opened a new log
I20260812 19:12:26.107879 21378 ts_tablet_manager.cc:1397] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa: Time spent bootstrapping tablet: real 0.188s user 0.014s sys 0.005s
I20260812 19:12:26.111511 21381 tablet_bootstrap.cc:654] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:26.124202 21734 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:26.124814 21734 raft_consensus.cc:504] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.129617 21378 raft_consensus.cc:348] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.130466 21378 raft_consensus.cc:374] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:26.130771 21378 raft_consensus.cc:729] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Initialized, Role: FOLLOWER
I20260812 19:12:26.131496 21378 consensus_queue.cc:260] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.133144 21378 ts_tablet_manager.cc:1428] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260812 19:12:26.134260 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:26.126863 21734 leader_election.cc:288] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:26.141847 21379 tablet_bootstrap.cc:492] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:26.142482 21379 ts_tablet_manager.cc:1397] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.132s user 0.005s sys 0.007s
I20260812 19:12:26.152561 21381 tablet_bootstrap.cc:492] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:26.153178 21381 ts_tablet_manager.cc:1397] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.165s user 0.008s sys 0.012s
I20260812 19:12:26.155326 21037 catalog_manager.cc:5495] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:26.156073 21381 raft_consensus.cc:348] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.160578 21381 raft_consensus.cc:374] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:26.160877 21381 raft_consensus.cc:729] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:26.161607 21381 consensus_queue.cc:260] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.159674 21379 raft_consensus.cc:348] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.163430 21379 raft_consensus.cc:374] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:26.163722 21379 raft_consensus.cc:729] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:26.164530 21379 consensus_queue.cc:260] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.165409 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:26.166108 21241 raft_consensus.cc:2454] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:26.167200 21381 ts_tablet_manager.cc:1428] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.014s user 0.008s sys 0.002s
I20260812 19:12:26.169090 21379 ts_tablet_manager.cc:1428] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260812 19:12:26.170151 21379 tablet_bootstrap.cc:492] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:26.171428 21947 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:26.173828 21381 tablet_bootstrap.cc:492] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:26.173734 21947 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.177774 21947 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:12:26.170269 21287 leader_election.cc:341] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:26.180464 21381 tablet_bootstrap.cc:654] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:26.186204 21379 tablet_bootstrap.cc:654] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:26.186743 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:26.187532 21151 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:26.193426 21381 tablet_bootstrap.cc:492] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:26.193229 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:26.193964 21381 ts_tablet_manager.cc:1397] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.002s
I20260812 19:12:26.197386 21381 raft_consensus.cc:348] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.198148 21381 raft_consensus.cc:374] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:26.198415 21381 raft_consensus.cc:729] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:26.193974 21241 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:26.199203 21381 consensus_queue.cc:260] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.201818 21381 ts_tablet_manager.cc:1428] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260812 19:12:26.202912 21381 tablet_bootstrap.cc:492] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:26.209245 21381 tablet_bootstrap.cc:654] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:26.219811 21287 leader_election.cc:302] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:26.221228 21947 raft_consensus.cc:2781] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:26.221860 21947 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:26.222294 21947 raft_consensus.cc:3037] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:26.222883 21891 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:12:26.234939 21287 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:26.239225 21734 raft_consensus.cc:2781] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:26.239679 21734 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:26.240032 21734 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:26.264998 21379 tablet_bootstrap.cc:492] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:26.265566 21379 ts_tablet_manager.cc:1397] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.096s user 0.013s sys 0.024s
I20260812 19:12:26.268085 21379 raft_consensus.cc:348] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.268373 21891 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:12:26.268857 21379 raft_consensus.cc:374] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:26.269152 21379 raft_consensus.cc:729] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:26.269845 21379 consensus_queue.cc:260] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.274359 21734 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.280055 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:26.280777 21241 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:26.282395 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:26.283238 21151 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:26.288589 21241 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:26.289744 21947 raft_consensus.cc:504] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.291162 21287 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:26.294984 21381 tablet_bootstrap.cc:492] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:26.295616 21381 ts_tablet_manager.cc:1397] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.093s user 0.028s sys 0.037s
I20260812 19:12:26.278393 21734 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:26.300427 21839 raft_consensus.cc:2781] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:26.302155 21685 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:26.307685 21151 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:26.313004 21947 leader_election.cc:288] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:26.311233 21685 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.315874 21839 raft_consensus.cc:686] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:26.317404 21839 consensus_queue.cc:237] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.324019 21379 ts_tablet_manager.cc:1428] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.058s user 0.008s sys 0.000s
I20260812 19:12:26.327749 21379 tablet_bootstrap.cc:492] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412: Bootstrap starting.
I20260812 19:12:26.325572 21685 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:26.328581 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:26.329739 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:26.330440 21152 raft_consensus.cc:3037] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:26.329355 21241 raft_consensus.cc:3037] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:26.357438 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:26.372965 21379 tablet_bootstrap.cc:654] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:26.378463 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:26.379326 21243 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:26.383064 21381 raft_consensus.cc:348] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.383957 21381 raft_consensus.cc:374] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:26.384258 21381 raft_consensus.cc:729] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:26.394050 21381 consensus_queue.cc:260] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.388638 21152 raft_consensus.cc:2454] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:26.401376 21241 raft_consensus.cc:2454] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:26.408208 21287 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:26.409395 21947 raft_consensus.cc:2781] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:26.411218 21947 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:26.411710 21947 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:26.412977 21150 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:26.425482 21381 ts_tablet_manager.cc:1428] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.129s user 0.008s sys 0.010s
I20260812 19:12:26.433094 21381 tablet_bootstrap.cc:492] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:26.440111 21034 catalog_manager.cc:5495] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:26.455133 21381 tablet_bootstrap.cc:654] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:26.440019 21287 leader_election.cc:302] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:26.456735 21947 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.463335 21834 raft_consensus.cc:2781] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:26.463574 21947 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:26.455448 21379 tablet_bootstrap.cc:492] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412: No bootstrap required, opened a new log
I20260812 19:12:26.467195 21379 ts_tablet_manager.cc:1397] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412: Time spent bootstrapping tablet: real 0.140s user 0.012s sys 0.018s
I20260812 19:12:26.468597 21834 raft_consensus.cc:686] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
W20260812 19:12:26.470866 21287 consensus_peers.cc:487] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 -> Peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Couldn't send request to peer f741102bfe1441d1b453e5a838649059. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:26.471174 21834 consensus_queue.cc:237] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.471086 21379 raft_consensus.cc:348] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.477219 21379 raft_consensus.cc:374] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:26.477684 21379 raft_consensus.cc:729] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Initialized, Role: FOLLOWER
I20260812 19:12:26.516826 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:26.518659 21158 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:26.527230 21379 consensus_queue.cc:260] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.536612 21834 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:26.537904 21834 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.539954 21834 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:26.540135 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:26.542371 21239 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:26.548648 21158 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:26.550380 21379 ts_tablet_manager.cc:1428] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412: Time spent starting tablet: real 0.083s user 0.011s sys 0.001s
I20260812 19:12:26.556852 21239 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:26.559307 21287 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:26.560520 21034 catalog_manager.cc:5495] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:26.563058 21685 raft_consensus.cc:2781] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:26.563859 21685 raft_consensus.cc:686] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:26.564371 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:26.565049 21685 consensus_queue.cc:237] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:26.572052 21287 leader_election.cc:341] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:26.579231 21381 tablet_bootstrap.cc:492] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:26.579950 21381 ts_tablet_manager.cc:1397] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.152s user 0.040s sys 0.047s
I20260812 19:12:26.586828 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:26.587600 21233 raft_consensus.cc:2454] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 0.
I20260812 19:12:26.590461 21381 raft_consensus.cc:348] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.593935 21034 catalog_manager.cc:5495] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:26.597355 21381 raft_consensus.cc:374] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:26.598852 21381 raft_consensus.cc:729] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
W20260812 19:12:26.595593 21287 consensus_peers.cc:487] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 -> Peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Couldn't send request to peer f741102bfe1441d1b453e5a838649059. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:26.602818 21381 consensus_queue.cc:260] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.622656 21381 ts_tablet_manager.cc:1428] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.042s user 0.009s sys 0.007s
I20260812 19:12:26.623826 21287 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:26.624706 21685 raft_consensus.cc:2781] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:26.625088 21685 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:26.625432 21685 raft_consensus.cc:3037] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:26.625847 21839 consensus_queue.cc:1035] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:12:26.628393 21883 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:26.628968 21883 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.633488 21381 tablet_bootstrap.cc:492] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
I20260812 19:12:26.643322 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:26.642062 21883 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:26.644166 21239 raft_consensus.cc:2454] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:26.644428 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:26.645095 21319 raft_consensus.cc:2454] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 0.
I20260812 19:12:26.646266 21112 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:26.647511 21825 raft_consensus.cc:2781] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 19:12:26.648027 21825 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:26.648330 21825 raft_consensus.cc:3037] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:26.656055 21381 tablet_bootstrap.cc:654] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:26.667877 21734 consensus_queue.cc:1035] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:26.650976 21685 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.681023 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:12:26.682801 21287 leader_election.cc:341] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Illegal state: must be running to vote when last-logged opid is not known
I20260812 19:12:26.685928 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:26.690347 21243 raft_consensus.cc:3037] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:26.679924 21685 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:26.711388 21825 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.719823 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:26.719799 21825 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:26.720676 21327 raft_consensus.cc:3037] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:26.721351 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:26.722007 21239 raft_consensus.cc:3037] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:26.730327 21239 raft_consensus.cc:2454] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:26.732324 21114 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:26.733372 21881 raft_consensus.cc:2781] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:26.733937 21881 raft_consensus.cc:686] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:26.734962 21881 consensus_queue.cc:237] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.753859 21381 tablet_bootstrap.cc:492] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:26.754489 21381 ts_tablet_manager.cc:1397] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.121s user 0.012s sys 0.022s
I20260812 19:12:26.762178 21381 raft_consensus.cc:348] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.763180 21381 raft_consensus.cc:374] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:26.763615 21381 raft_consensus.cc:729] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:26.764967 21381 consensus_queue.cc:260] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.769384 21243 raft_consensus.cc:2454] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:12:26.771081 21287 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:12:26.774263 21327 raft_consensus.cc:2454] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:12:26.778940 21839 raft_consensus.cc:2781] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 19:12:26.779549 21839 raft_consensus.cc:686] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:26.780514 21839 consensus_queue.cc:237] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.791488 21034 catalog_manager.cc:5495] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:26.799651 21381 ts_tablet_manager.cc:1428] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.045s user 0.007s sys 0.000s
I20260812 19:12:26.803231 21381 tablet_bootstrap.cc:492] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059: Bootstrap starting.
W20260812 19:12:26.815037 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 48867.1us
I20260812 19:12:26.822270 21881 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:26.822860 21881 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.828907 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:26.829620 21381 tablet_bootstrap.cc:654] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059: Neither blocks nor log segments found. Creating new log.
I20260812 19:12:26.861155 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:26.863384 21112 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:26.866250 21685 consensus_queue.cc:1035] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:26.867954 21034 catalog_manager.cc:5495] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:26.869185 21825 raft_consensus.cc:2726] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:26.886240 21881 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:26.889058 21381 tablet_bootstrap.cc:492] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059: No bootstrap required, opened a new log
I20260812 19:12:26.889597 21381 ts_tablet_manager.cc:1397] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059: Time spent bootstrapping tablet: real 0.089s user 0.025s sys 0.034s
I20260812 19:12:26.891055 21839 consensus_queue.cc:1035] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:26.892212 21381 raft_consensus.cc:348] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.893008 21381 raft_consensus.cc:374] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 19:12:26.893306 21381 raft_consensus.cc:729] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Initialized, Role: FOLLOWER
I20260812 19:12:26.893963 21381 consensus_queue.cc:260] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:26.896508 21381 ts_tablet_manager.cc:1428] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 19:12:26.938905 21839 consensus_queue.cc:1035] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:12:26.951428 21839 consensus_queue.cc:1035] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:26.964805 21150 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:26.988915 21151 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:27.004113 21839 consensus_queue.cc:1035] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:27.027748 21839 consensus_queue.cc:1035] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:27.059849 21976 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:27.060422 21976 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:27.066372 21976 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:27.068557 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:27.067719 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:27.073748 21239 raft_consensus.cc:2379] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 1: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:12:27.075286 21114 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:27.076757 21881 raft_consensus.cc:3037] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 0 FOLLOWER]: Advancing to term 1
I20260812 19:12:27.086143 21881 raft_consensus.cc:2726] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 19:12:27.185406 21976 consensus_queue.cc:1035] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:12:27.187665 21976 consensus_queue.cc:1035] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:12:27.235042 21834 consensus_queue.cc:1035] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 19:12:27.251308 21805 consensus_queue.cc:1035] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 19:12:27.499087 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 19:12:27.507710 21995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:12:27.524762 21996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 19:12:27.526472 22000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 19:12:27.529135 19738 server_base.cc:1034] running on GCE node
I20260812 19:12:27.531256 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 19:12:27.531634 19738 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 19:12:27.531867 19738 hybrid_clock.cc:648] HybridClock initialized: now 1786561947531850 us; error 0 us; skew 500 ppm
I20260812 19:12:27.532471 19738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 19:12:27.536312 19738 webserver.cc:460] Webserver started at http://127.19.70.132:41001/ using document root <none> and password file <none>
I20260812 19:12:27.536949 19738 fs_manager.cc:359] Metadata directory not provided
I20260812 19:12:27.537180 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 19:12:27.537593 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 19:12:27.539465 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-3-root/instance:
uuid: "63db10063a7d4eb2a9cc9dc952654706"
format_stamp: "Formatted at 2026-08-12 19:12:27 on dist-test-slave-pgkr"
I20260812 19:12:27.544955 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 19:12:27.548800 22007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 19:12:27.549845 19738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 19:12:27.550241 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-3-root
uuid: "63db10063a7d4eb2a9cc9dc952654706"
format_stamp: "Formatted at 2026-08-12 19:12:27 on dist-test-slave-pgkr"
I20260812 19:12:27.550544 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 19:12:27.573652 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 19:12:27.575502 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 19:12:27.577172 19738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 19:12:27.580950 19738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 19:12:27.581296 19738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 19:12:27.581625 19738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 19:12:27.581864 19738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 19:12:27.737818 19738 rpc_server.cc:307] RPC server started. Bound to: 127.19.70.132:39579
I20260812 19:12:27.738027 22084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.70.132:39579 every 8 connection(s)
I20260812 19:12:27.744772 19738 auto_leader_rebalancer-test.cc:413] add a tserver: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:27.766578 22085 heartbeater.cc:347] Connected to a master server at 127.19.70.190:35311
I20260812 19:12:27.767166 22085 heartbeater.cc:464] Registering TS with master...
I20260812 19:12:27.767992 22085 heartbeater.cc:511] Master 127.19.70.190:35311 requested a full tablet report, sending...
I20260812 19:12:27.770331 21034 ts_manager.cc:194] Registered new tserver with Master: 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:12:27.772468 21034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58998
I20260812 19:12:28.132591 21243 consensus_queue.cc:237] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:28.147444 21327 raft_consensus.cc:1261] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:28.148846 21826 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:28.149575 21151 raft_consensus.cc:1261] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:28.156509 21826 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:28.174479 21891 raft_consensus.cc:2932] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
W20260812 19:12:28.176748 21204 consensus_peers.cc:487] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8161580b52441ab9a17a0e1597f243b. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:28.184043 21327 raft_consensus.cc:2932] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:28.190510 21151 raft_consensus.cc:2932] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:28.226657 21034 catalog_manager.cc:5495] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:28.296142 21327 consensus_queue.cc:237] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:28.305706 21151 raft_consensus.cc:1261] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:28.305706 21243 raft_consensus.cc:1261] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:28.307416 21970 consensus_queue.cc:1035] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:28.309655 21970 consensus_queue.cc:1035] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:28.327082 21839 raft_consensus.cc:2932] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
W20260812 19:12:28.329558 21288 consensus_peers.cc:487] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b730f70f25e440e68595cc248b40ee98. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:28.331111 21151 raft_consensus.cc:2932] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:28.335162 21241 raft_consensus.cc:2932] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:28.345853 21243 consensus_queue.cc:237] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:28.353214 21035 catalog_manager.cc:5495] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
W20260812 19:12:28.358206 21204 consensus_peers.cc:487] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7e49dcb1cac44ee91df33a2abea1dd0. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:28.358351 21327 raft_consensus.cc:1261] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:28.360491 21891 consensus_queue.cc:1035] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:28.364917 21151 raft_consensus.cc:1261] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:28.370723 22099 consensus_queue.cc:1035] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:28.381958 21319 raft_consensus.cc:2932] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:28.380739 22098 raft_consensus.cc:2932] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:28.401508 21152 raft_consensus.cc:2932] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:28.408869 21320 consensus_queue.cc:237] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:28.423005 21034 catalog_manager.cc:5495] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:28.443210 21241 raft_consensus.cc:1261] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:28.445700 21970 consensus_queue.cc:1035] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260812 19:12:28.458106 21288 consensus_peers.cc:487] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a070b29d02d24ef4b36656fcac6d68d9. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:28.458454 21152 raft_consensus.cc:1261] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:28.461205 21970 consensus_queue.cc:1035] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:28.467412 21970 raft_consensus.cc:2932] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:28.468775 21243 raft_consensus.cc:2932] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:28.474967 21151 raft_consensus.cc:2932] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:28.491720 21034 catalog_manager.cc:5495] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:28.636795 22117 ts_tablet_manager.cc:927] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:28.639024 22117 tablet_copy_client.cc:316] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.130:33807
I20260812 19:12:28.662346 21255 tablet_copy_service.cc:140] P d1e1da0b9585460e8c365bf55679d4fa: Received BeginTabletCopySession request for tablet a8161580b52441ab9a17a0e1597f243b from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:57682)
I20260812 19:12:28.663043 21255 tablet_copy_service.cc:161] P d1e1da0b9585460e8c365bf55679d4fa: Beginning new tablet copy session on tablet a8161580b52441ab9a17a0e1597f243b from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:57682: session id = 63db10063a7d4eb2a9cc9dc952654706-a8161580b52441ab9a17a0e1597f243b
I20260812 19:12:28.671775 21255 tablet_copy_source_session.cc:205] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:28.676728 22117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8161580b52441ab9a17a0e1597f243b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:28.694522 22117 tablet_copy_client.cc:791] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:28.695277 22117 tablet_copy_client.cc:655] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:28.700029 22117 tablet_copy_client.cc:531] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:28.710081 22117 tablet_bootstrap.cc:492] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:28.743221 22117 tablet_bootstrap.cc:492] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:28.744444 22117 tablet_bootstrap.cc:492] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:28.745302 22117 ts_tablet_manager.cc:1397] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.036s user 0.016s sys 0.016s
I20260812 19:12:28.748842 22117 raft_consensus.cc:348] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:28.749871 22117 raft_consensus.cc:729] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:28.750723 22117 consensus_queue.cc:260] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:28.760046 22085 heartbeater.cc:503] Master 127.19.70.190:35311 was elected leader, sending a full tablet report...
I20260812 19:12:28.762753 22117 ts_tablet_manager.cc:1428] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.017s user 0.012s sys 0.005s
I20260812 19:12:28.767390 21255 tablet_copy_service.cc:342] P d1e1da0b9585460e8c365bf55679d4fa: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-a8161580b52441ab9a17a0e1597f243b received from {username='slave'} at 127.0.0.1:57682
I20260812 19:12:28.767870 21255 tablet_copy_service.cc:433] P d1e1da0b9585460e8c365bf55679d4fa: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-a8161580b52441ab9a17a0e1597f243b on tablet a8161580b52441ab9a17a0e1597f243b with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:28.801707 22117 ts_tablet_manager.cc:927] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:28.803869 22117 tablet_copy_client.cc:316] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.130:33807
I20260812 19:12:28.805594 21255 tablet_copy_service.cc:140] P d1e1da0b9585460e8c365bf55679d4fa: Received BeginTabletCopySession request for tablet e7e49dcb1cac44ee91df33a2abea1dd0 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:57682)
I20260812 19:12:28.806332 21255 tablet_copy_service.cc:161] P d1e1da0b9585460e8c365bf55679d4fa: Beginning new tablet copy session on tablet e7e49dcb1cac44ee91df33a2abea1dd0 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:57682: session id = 63db10063a7d4eb2a9cc9dc952654706-e7e49dcb1cac44ee91df33a2abea1dd0
I20260812 19:12:28.815203 21255 tablet_copy_source_session.cc:205] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:28.818610 22117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e49dcb1cac44ee91df33a2abea1dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:28.837399 22117 tablet_copy_client.cc:791] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:28.838055 22117 tablet_copy_client.cc:655] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:28.841744 22117 tablet_copy_client.cc:531] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:28.850858 22117 tablet_bootstrap.cc:492] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:28.868151 22127 ts_tablet_manager.cc:927] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:28.870796 22127 tablet_copy_client.cc:316] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.131:33285
I20260812 19:12:28.878769 22117 tablet_bootstrap.cc:492] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:28.880067 22117 tablet_bootstrap.cc:492] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:28.880853 22117 ts_tablet_manager.cc:1397] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.012s
I20260812 19:12:28.883805 22117 raft_consensus.cc:348] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:28.884755 22117 raft_consensus.cc:729] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:28.885429 22117 consensus_queue.cc:260] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:28.887890 22117 ts_tablet_manager.cc:1428] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260812 19:12:28.890007 21255 tablet_copy_service.cc:342] P d1e1da0b9585460e8c365bf55679d4fa: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-e7e49dcb1cac44ee91df33a2abea1dd0 received from {username='slave'} at 127.0.0.1:57682
I20260812 19:12:28.890515 21255 tablet_copy_service.cc:433] P d1e1da0b9585460e8c365bf55679d4fa: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-e7e49dcb1cac44ee91df33a2abea1dd0 on tablet e7e49dcb1cac44ee91df33a2abea1dd0 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:28.894013 21342 tablet_copy_service.cc:140] P 93578836968649f6bfcaecc709871412: Received BeginTabletCopySession request for tablet b730f70f25e440e68595cc248b40ee98 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:50042)
I20260812 19:12:28.894776 21342 tablet_copy_service.cc:161] P 93578836968649f6bfcaecc709871412: Beginning new tablet copy session on tablet b730f70f25e440e68595cc248b40ee98 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:50042: session id = 63db10063a7d4eb2a9cc9dc952654706-b730f70f25e440e68595cc248b40ee98
I20260812 19:12:28.908805 21342 tablet_copy_source_session.cc:205] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:28.970358 22127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b730f70f25e440e68595cc248b40ee98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:28.995249 22117 ts_tablet_manager.cc:927] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:12:29.103814 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 58815.3us
I20260812 19:12:29.118942 22117 tablet_copy_client.cc:316] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.131:33285
I20260812 19:12:29.142148 21342 tablet_copy_service.cc:140] P 93578836968649f6bfcaecc709871412: Received BeginTabletCopySession request for tablet a070b29d02d24ef4b36656fcac6d68d9 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:50042)
I20260812 19:12:29.142949 21342 tablet_copy_service.cc:161] P 93578836968649f6bfcaecc709871412: Beginning new tablet copy session on tablet a070b29d02d24ef4b36656fcac6d68d9 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:50042: session id = 63db10063a7d4eb2a9cc9dc952654706-a070b29d02d24ef4b36656fcac6d68d9
I20260812 19:12:29.160399 21342 tablet_copy_source_session.cc:205] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:29.173676 22117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a070b29d02d24ef4b36656fcac6d68d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:29.174868 22127 tablet_copy_client.cc:791] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:29.179049 22127 tablet_copy_client.cc:655] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:29.217267 22127 tablet_copy_client.cc:531] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:29.230922 22117 tablet_copy_client.cc:791] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:29.231760 22117 tablet_copy_client.cc:655] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:29.233950 22127 tablet_bootstrap.cc:492] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:29.237421 22117 tablet_copy_client.cc:531] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:29.256958 22117 tablet_bootstrap.cc:492] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:29.276819 22127 tablet_bootstrap.cc:492] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:29.279393 22056 raft_consensus.cc:1203] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:29.281458 22117 tablet_bootstrap.cc:492] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:29.282512 22117 tablet_bootstrap.cc:492] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:29.283284 22117 ts_tablet_manager.cc:1397] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.027s user 0.019s sys 0.004s
I20260812 19:12:29.278087 22127 tablet_bootstrap.cc:492] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:29.286371 22127 ts_tablet_manager.cc:1397] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.053s user 0.026s sys 0.007s
I20260812 19:12:29.286777 22117 raft_consensus.cc:348] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:29.288412 22117 raft_consensus.cc:729] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:29.289173 22117 consensus_queue.cc:260] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:29.291750 22117 ts_tablet_manager.cc:1428] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20260812 19:12:29.294494 21342 tablet_copy_service.cc:342] P 93578836968649f6bfcaecc709871412: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-a070b29d02d24ef4b36656fcac6d68d9 received from {username='slave'} at 127.0.0.1:50042
I20260812 19:12:29.295042 21342 tablet_copy_service.cc:433] P 93578836968649f6bfcaecc709871412: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-a070b29d02d24ef4b36656fcac6d68d9 on tablet a070b29d02d24ef4b36656fcac6d68d9 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:29.297255 22127 raft_consensus.cc:348] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:29.298435 22127 raft_consensus.cc:729] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:29.299234 22127 consensus_queue.cc:260] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:29.301748 22127 ts_tablet_manager.cc:1428] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.015s user 0.001s sys 0.004s
I20260812 19:12:29.305621 21342 tablet_copy_service.cc:342] P 93578836968649f6bfcaecc709871412: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-b730f70f25e440e68595cc248b40ee98 received from {username='slave'} at 127.0.0.1:50042
I20260812 19:12:29.306252 21342 tablet_copy_service.cc:433] P 93578836968649f6bfcaecc709871412: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-b730f70f25e440e68595cc248b40ee98 on tablet b730f70f25e440e68595cc248b40ee98 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:29.372366 22056 raft_consensus.cc:1203] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:29.381394 22056 raft_consensus.cc:1203] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:29.487725 22056 raft_consensus.cc:1203] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:29.671506 22098 raft_consensus.cc:1050] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:29.674841 22098 consensus_queue.cc:237] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:29.683408 22056 raft_consensus.cc:1261] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:29.683802 21328 raft_consensus.cc:1261] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:29.685009 21151 raft_consensus.cc:1261] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:29.685245 21891 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:29.687050 22098 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:12:29.689098 22138 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:29.722707 22099 raft_consensus.cc:2932] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:29.725989 21151 raft_consensus.cc:2932] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:29.751811 21034 catalog_manager.cc:5495] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:29.754312 22056 raft_consensus.cc:2932] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:29.758272 21328 raft_consensus.cc:2932] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:29.785256 21834 raft_consensus.cc:1050] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:29.788210 21834 consensus_queue.cc:237] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:29.792371 21891 raft_consensus.cc:1050] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:29.795511 21891 consensus_queue.cc:237] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:29.798055 22056 raft_consensus.cc:1261] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:29.807245 21834 consensus_queue.cc:1035] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:29.813660 21151 raft_consensus.cc:1261] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:29.818923 21241 raft_consensus.cc:1261] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:29.821985 21834 consensus_queue.cc:1035] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:29.826170 21970 consensus_queue.cc:1035] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:29.836591 21233 consensus_queue.cc:237] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:29.874200 22055 raft_consensus.cc:1261] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:29.877134 21891 consensus_queue.cc:1035] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:29.888105 21319 raft_consensus.cc:1261] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:29.890834 21891 consensus_queue.cc:1035] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:29.901026 21149 raft_consensus.cc:1261] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:29.900171 21970 raft_consensus.cc:2932] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:29.908260 22138 consensus_queue.cc:1035] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:29.908918 22056 raft_consensus.cc:1261] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:29.912858 22138 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:29.913169 21320 raft_consensus.cc:1261] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:29.927446 22138 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:12:29.929742 22161 raft_consensus.cc:2932] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:29.952191 21233 raft_consensus.cc:2932] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:29.946089 21151 raft_consensus.cc:2932] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:29.963984 22056 raft_consensus.cc:2932] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:29.970137 22160 raft_consensus.cc:1050] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:29.973726 22160 consensus_queue.cc:237] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:29.976059 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a8161580b52441ab9a17a0e1597f243b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:29.981002 21036 catalog_manager.cc:5495] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:29.941059 21319 raft_consensus.cc:2932] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:29.986204 21037 catalog_manager.cc:5495] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:30.089651 22055 raft_consensus.cc:2932] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.092183 22099 raft_consensus.cc:2932] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.109556 22056 raft_consensus.cc:2932] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.125252 21151 raft_consensus.cc:2932] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.131631 22055 raft_consensus.cc:1261] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:30.153157 21320 raft_consensus.cc:2932] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.156222 22160 consensus_queue.cc:1035] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260812 19:12:30.183321 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet a8161580b52441ab9a17a0e1597f243b with delete_type TABLET_DATA_TOMBSTONED (TS f741102bfe1441d1b453e5a838649059 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54786
W20260812 19:12:30.191684 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 214242us
I20260812 19:12:30.197252 21035 catalog_manager.cc:5495] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.204553 21241 raft_consensus.cc:1261] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:30.213548 22137 consensus_queue.cc:1035] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:30.219753 21151 raft_consensus.cc:1261] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:30.291280 22137 consensus_queue.cc:1035] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:30.299121 22177 tablet_replica.cc:330] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059: stopping tablet replica
I20260812 19:12:30.307821 21239 consensus_queue.cc:237] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:30.307826 21322 consensus_queue.cc:237] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:30.316537 22056 raft_consensus.cc:1261] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:30.316391 22177 raft_consensus.cc:2229] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:30.321126 22098 consensus_queue.cc:1035] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:30.322654 22056 raft_consensus.cc:1261] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:30.326655 22160 consensus_queue.cc:1035] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:30.335187 22163 raft_consensus.cc:2932] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.330355 22177 raft_consensus.cc:2258] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:30.335958 22056 raft_consensus.cc:2932] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.348966 22165 raft_consensus.cc:2932] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.355624 22053 raft_consensus.cc:2932] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.357352 21036 catalog_manager.cc:5495] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:30.363478 21019 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b730f70f25e440e68595cc248b40ee98 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:30.367519 21151 raft_consensus.cc:1261] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:30.371066 22177 ts_tablet_manager.cc:1905] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:30.377382 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e7e49dcb1cac44ee91df33a2abea1dd0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:30.379878 21151 raft_consensus.cc:1261] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:30.376405 22098 consensus_queue.cc:1035] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260812 19:12:30.381989 22137 raft_consensus.cc:2932] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.386659 21034 catalog_manager.cc:5495] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:30.388036 21316 tablet_service.cc:1514] Processing DeleteTablet for tablet e7e49dcb1cac44ee91df33a2abea1dd0 with delete_type TABLET_DATA_TOMBSTONED (TS 93578836968649f6bfcaecc709871412 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51536
I20260812 19:12:30.409996 22056 raft_consensus.cc:2932] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.417344 21839 consensus_queue.cc:1035] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:30.413615 21158 raft_consensus.cc:2932] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.431898 21219 tablet_service.cc:1514] Processing DeleteTablet for tablet b730f70f25e440e68595cc248b40ee98 with delete_type TABLET_DATA_TOMBSTONED (TS d1e1da0b9585460e8c365bf55679d4fa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37890
I20260812 19:12:30.432446 21151 raft_consensus.cc:2932] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.441610 21235 raft_consensus.cc:2932] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.446933 22181 tablet_replica.cc:330] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412: stopping tablet replica
I20260812 19:12:30.451565 22181 raft_consensus.cc:2229] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:30.452814 22181 raft_consensus.cc:2258] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:30.458803 22181 ts_tablet_manager.cc:1905] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:30.457192 21037 catalog_manager.cc:5495] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.462687 21157 raft_consensus.cc:2932] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.502068 22177 ts_tablet_manager.cc:1918] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:30.502919 22177 log.cc:1196] T a8161580b52441ab9a17a0e1597f243b P f741102bfe1441d1b453e5a838649059: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-0-root/wals/a8161580b52441ab9a17a0e1597f243b
I20260812 19:12:30.517702 22182 tablet_replica.cc:330] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa: stopping tablet replica
I20260812 19:12:30.517263 21320 consensus_queue.cc:237] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:30.524040 22182 raft_consensus.cc:2229] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:30.527135 22182 raft_consensus.cc:2258] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:30.529286 21022 catalog_manager.cc:4841] TS f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): tablet a8161580b52441ab9a17a0e1597f243b (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:30.534354 22182 ts_tablet_manager.cc:1905] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:30.538151 22056 raft_consensus.cc:1261] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:30.540128 21158 raft_consensus.cc:1261] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:30.542665 22137 consensus_queue.cc:1035] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:30.544937 22137 consensus_queue.cc:1035] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:12:30.545189 22181 ts_tablet_manager.cc:1918] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:30.545882 22181 log.cc:1196] T e7e49dcb1cac44ee91df33a2abea1dd0 P 93578836968649f6bfcaecc709871412: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-2-root/wals/e7e49dcb1cac44ee91df33a2abea1dd0
I20260812 19:12:30.549767 21019 catalog_manager.cc:4841] TS 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): tablet e7e49dcb1cac44ee91df33a2abea1dd0 (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:30.568358 21157 raft_consensus.cc:2932] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.576508 22182 ts_tablet_manager.cc:1918] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:30.577168 22182 log.cc:1196] T b730f70f25e440e68595cc248b40ee98 P d1e1da0b9585460e8c365bf55679d4fa: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-1-root/wals/b730f70f25e440e68595cc248b40ee98
I20260812 19:12:30.582927 21022 catalog_manager.cc:4841] TS d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): tablet b730f70f25e440e68595cc248b40ee98 (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:30.587437 21834 raft_consensus.cc:2932] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.589838 22056 raft_consensus.cc:2932] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.605880 21034 catalog_manager.cc:5495] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:30.618887 21019 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a070b29d02d24ef4b36656fcac6d68d9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:30.639050 21219 tablet_service.cc:1514] Processing DeleteTablet for tablet a070b29d02d24ef4b36656fcac6d68d9 with delete_type TABLET_DATA_TOMBSTONED (TS d1e1da0b9585460e8c365bf55679d4fa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37890
I20260812 19:12:30.640198 22182 tablet_replica.cc:330] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa: stopping tablet replica
I20260812 19:12:30.641225 22182 raft_consensus.cc:2229] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:30.642076 22182 raft_consensus.cc:2258] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:30.647751 22182 ts_tablet_manager.cc:1905] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:30.678505 22182 ts_tablet_manager.cc:1918] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:30.679042 22182 log.cc:1196] T a070b29d02d24ef4b36656fcac6d68d9 P d1e1da0b9585460e8c365bf55679d4fa: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-1-root/wals/a070b29d02d24ef4b36656fcac6d68d9
I20260812 19:12:30.681193 21022 catalog_manager.cc:4841] TS d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): tablet a070b29d02d24ef4b36656fcac6d68d9 (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:31.708148 21239 consensus_queue.cc:237] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
W20260812 19:12:31.717718 21204 consensus_peers.cc:487] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f11f84389864fddb8350b0f94829938. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:31.721421 21158 raft_consensus.cc:1261] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:31.723371 21322 raft_consensus.cc:1261] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:31.727744 22138 consensus_queue.cc:1035] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:31.744956 22138 consensus_queue.cc:1035] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:31.757668 21322 raft_consensus.cc:2932] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:31.771678 22161 raft_consensus.cc:2932] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:31.798457 21158 raft_consensus.cc:2932] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:31.816560 21036 catalog_manager.cc:5495] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:31.858750 21158 consensus_queue.cc:237] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:31.867630 21322 raft_consensus.cc:1261] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:31.870267 22174 consensus_queue.cc:1035] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:31.889199 21233 raft_consensus.cc:1261] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:31.891086 22174 consensus_queue.cc:1035] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:31.887836 22183 raft_consensus.cc:2932] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:31.898965 21233 raft_consensus.cc:2932] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:31.908869 21322 raft_consensus.cc:2932] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
W20260812 19:12:31.917413 21115 consensus_peers.cc:487] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6521960ce6040af919a9e227d2211ec. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:31.923995 21037 catalog_manager.cc:5495] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:31.925766 21233 consensus_queue.cc:237] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:31.938961 21320 raft_consensus.cc:1261] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:31.940332 22185 consensus_queue.cc:1035] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:31.941013 21158 raft_consensus.cc:1261] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:31.942519 22138 consensus_queue.cc:1035] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260812 19:12:31.943508 21204 consensus_peers.cc:487] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae9be4809de24d32bdd888c6f9511d0f. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:31.958758 22185 raft_consensus.cc:2932] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:31.963945 21151 raft_consensus.cc:2932] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:31.967084 21320 raft_consensus.cc:2932] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:31.976024 21158 consensus_queue.cc:237] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:31.985273 21036 catalog_manager.cc:5495] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:31.989457 21320 raft_consensus.cc:1261] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 19:12:31.990541 21115 consensus_peers.cc:487] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aff4931ad1bb4657aa3650a15b43ddcd. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:31.991008 22183 consensus_queue.cc:1035] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:31.992206 21233 raft_consensus.cc:1261] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:31.994261 22183 consensus_queue.cc:1035] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:32.012117 22172 raft_consensus.cc:2932] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:32.015581 21322 raft_consensus.cc:2932] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:32.018344 21239 raft_consensus.cc:2932] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:32.031394 21036 catalog_manager.cc:5495] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:32.182832 22217 ts_tablet_manager.cc:927] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:32.188445 22217 tablet_copy_client.cc:316] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.130:33807
I20260812 19:12:32.190575 21255 tablet_copy_service.cc:140] P d1e1da0b9585460e8c365bf55679d4fa: Received BeginTabletCopySession request for tablet 3f11f84389864fddb8350b0f94829938 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:57682)
I20260812 19:12:32.191190 21255 tablet_copy_service.cc:161] P d1e1da0b9585460e8c365bf55679d4fa: Beginning new tablet copy session on tablet 3f11f84389864fddb8350b0f94829938 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:57682: session id = 63db10063a7d4eb2a9cc9dc952654706-3f11f84389864fddb8350b0f94829938
I20260812 19:12:32.199254 21255 tablet_copy_source_session.cc:205] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:32.203140 22217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f11f84389864fddb8350b0f94829938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:32.220516 22217 tablet_copy_client.cc:791] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:32.221346 22217 tablet_copy_client.cc:655] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:32.227252 22217 tablet_copy_client.cc:531] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:32.237308 22217 tablet_bootstrap.cc:492] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:32.260759 22217 tablet_bootstrap.cc:492] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:32.261597 22217 tablet_bootstrap.cc:492] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:32.262287 22217 ts_tablet_manager.cc:1397] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.025s user 0.017s sys 0.004s
I20260812 19:12:32.265236 22217 raft_consensus.cc:348] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:32.266223 22217 raft_consensus.cc:729] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:32.266916 22217 consensus_queue.cc:260] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:32.271512 22217 ts_tablet_manager.cc:1428] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.009s user 0.008s sys 0.004s
I20260812 19:12:32.273566 21255 tablet_copy_service.cc:342] P d1e1da0b9585460e8c365bf55679d4fa: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-3f11f84389864fddb8350b0f94829938 received from {username='slave'} at 127.0.0.1:57682
I20260812 19:12:32.274019 21255 tablet_copy_service.cc:433] P d1e1da0b9585460e8c365bf55679d4fa: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-3f11f84389864fddb8350b0f94829938 on tablet 3f11f84389864fddb8350b0f94829938 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:32.355907 22217 ts_tablet_manager.cc:927] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:32.378115 22217 tablet_copy_client.cc:316] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.130:33807
I20260812 19:12:32.380697 21255 tablet_copy_service.cc:140] P d1e1da0b9585460e8c365bf55679d4fa: Received BeginTabletCopySession request for tablet ae9be4809de24d32bdd888c6f9511d0f from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:57682)
I20260812 19:12:32.381350 21255 tablet_copy_service.cc:161] P d1e1da0b9585460e8c365bf55679d4fa: Beginning new tablet copy session on tablet ae9be4809de24d32bdd888c6f9511d0f from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:57682: session id = 63db10063a7d4eb2a9cc9dc952654706-ae9be4809de24d32bdd888c6f9511d0f
I20260812 19:12:32.389961 21255 tablet_copy_source_session.cc:205] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:32.393862 22217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9be4809de24d32bdd888c6f9511d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:32.414253 22217 tablet_copy_client.cc:791] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:32.415015 22217 tablet_copy_client.cc:655] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:32.419618 22217 tablet_copy_client.cc:531] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:32.420845 22223 ts_tablet_manager.cc:927] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551)
I20260812 19:12:32.423022 22223 tablet_copy_client.cc:316] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.129:36551
I20260812 19:12:32.431435 22217 tablet_bootstrap.cc:492] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:32.461562 21170 tablet_copy_service.cc:140] P f741102bfe1441d1b453e5a838649059: Received BeginTabletCopySession request for tablet d6521960ce6040af919a9e227d2211ec from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:51004)
I20260812 19:12:32.462205 21170 tablet_copy_service.cc:161] P f741102bfe1441d1b453e5a838649059: Beginning new tablet copy session on tablet d6521960ce6040af919a9e227d2211ec from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:51004: session id = 63db10063a7d4eb2a9cc9dc952654706-d6521960ce6040af919a9e227d2211ec
I20260812 19:12:32.470341 22217 tablet_bootstrap.cc:492] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:32.470968 22229 ts_tablet_manager.cc:927] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551)
I20260812 19:12:32.471654 22217 tablet_bootstrap.cc:492] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:32.472690 22217 ts_tablet_manager.cc:1397] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.042s user 0.024s sys 0.012s
I20260812 19:12:32.473163 22229 tablet_copy_client.cc:316] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.129:36551
I20260812 19:12:32.474319 21170 tablet_copy_source_session.cc:205] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:32.475698 21169 tablet_copy_service.cc:140] P f741102bfe1441d1b453e5a838649059: Received BeginTabletCopySession request for tablet aff4931ad1bb4657aa3650a15b43ddcd from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:51004)
I20260812 19:12:32.476393 21169 tablet_copy_service.cc:161] P f741102bfe1441d1b453e5a838649059: Beginning new tablet copy session on tablet aff4931ad1bb4657aa3650a15b43ddcd from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:51004: session id = 63db10063a7d4eb2a9cc9dc952654706-aff4931ad1bb4657aa3650a15b43ddcd
I20260812 19:12:32.477244 22217 raft_consensus.cc:348] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:32.478245 22217 raft_consensus.cc:729] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:32.478996 22217 consensus_queue.cc:260] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:32.481594 22217 ts_tablet_manager.cc:1428] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.008s user 0.001s sys 0.004s
I20260812 19:12:32.486155 21255 tablet_copy_service.cc:342] P d1e1da0b9585460e8c365bf55679d4fa: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-ae9be4809de24d32bdd888c6f9511d0f received from {username='slave'} at 127.0.0.1:57682
I20260812 19:12:32.486658 21255 tablet_copy_service.cc:433] P d1e1da0b9585460e8c365bf55679d4fa: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-ae9be4809de24d32bdd888c6f9511d0f on tablet ae9be4809de24d32bdd888c6f9511d0f with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:32.495808 21169 tablet_copy_source_session.cc:205] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:32.499511 22223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6521960ce6040af919a9e227d2211ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:32.500393 22229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aff4931ad1bb4657aa3650a15b43ddcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:32.516985 22229 tablet_copy_client.cc:791] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:32.517798 22229 tablet_copy_client.cc:655] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:32.523016 22229 tablet_copy_client.cc:531] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:32.524173 22223 tablet_copy_client.cc:791] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:32.524794 22223 tablet_copy_client.cc:655] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:32.531242 22223 tablet_copy_client.cc:531] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:32.541814 22223 tablet_bootstrap.cc:492] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:32.545770 22229 tablet_bootstrap.cc:492] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:32.573735 22223 tablet_bootstrap.cc:492] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:32.574863 22223 tablet_bootstrap.cc:492] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:32.575698 22223 ts_tablet_manager.cc:1397] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.008s
I20260812 19:12:32.578764 22223 raft_consensus.cc:348] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:32.579734 22229 tablet_bootstrap.cc:492] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:32.579739 22223 raft_consensus.cc:729] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:32.580878 22229 tablet_bootstrap.cc:492] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:32.580602 22223 consensus_queue.cc:260] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:32.583380 22223 ts_tablet_manager.cc:1428] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.007s user 0.003s sys 0.005s
I20260812 19:12:32.583441 22229 ts_tablet_manager.cc:1397] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.038s user 0.021s sys 0.008s
I20260812 19:12:32.585820 21169 tablet_copy_service.cc:342] P f741102bfe1441d1b453e5a838649059: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-d6521960ce6040af919a9e227d2211ec received from {username='slave'} at 127.0.0.1:51004
I20260812 19:12:32.586251 21169 tablet_copy_service.cc:433] P f741102bfe1441d1b453e5a838649059: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-d6521960ce6040af919a9e227d2211ec on tablet d6521960ce6040af919a9e227d2211ec with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:32.586547 22229 raft_consensus.cc:348] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:32.587528 22229 raft_consensus.cc:729] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:32.588272 22229 consensus_queue.cc:260] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:32.590936 22229 ts_tablet_manager.cc:1428] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260812 19:12:32.593305 21169 tablet_copy_service.cc:342] P f741102bfe1441d1b453e5a838649059: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-aff4931ad1bb4657aa3650a15b43ddcd received from {username='slave'} at 127.0.0.1:51004
I20260812 19:12:32.593683 21169 tablet_copy_service.cc:433] P f741102bfe1441d1b453e5a838649059: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-aff4931ad1bb4657aa3650a15b43ddcd on tablet aff4931ad1bb4657aa3650a15b43ddcd with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:32.745414 22056 raft_consensus.cc:1203] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:32.765578 22056 raft_consensus.cc:1203] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:32.859602 22056 raft_consensus.cc:1203] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:32.899345 22056 raft_consensus.cc:1203] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:33.127843 22138 raft_consensus.cc:1050] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:33.129858 22138 consensus_queue.cc:237] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:33.135667 21320 raft_consensus.cc:1261] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:33.135669 22056 raft_consensus.cc:1261] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:33.137607 22188 consensus_queue.cc:1035] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:12:33.138320 21149 raft_consensus.cc:1261] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:33.140273 22138 consensus_queue.cc:1035] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:33.141322 22185 consensus_queue.cc:1035] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:12:33.156177 22188 raft_consensus.cc:2932] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.158587 21320 raft_consensus.cc:2932] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.161662 21150 raft_consensus.cc:2932] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.173688 22056 raft_consensus.cc:2932] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.178946 21034 catalog_manager.cc:5495] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.234918 21239 consensus_queue.cc:237] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:33.245353 22056 raft_consensus.cc:1261] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:33.247221 21320 raft_consensus.cc:1261] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:33.249830 22185 raft_consensus.cc:1050] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:33.254489 22185 consensus_queue.cc:237] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:33.255424 22138 consensus_queue.cc:1035] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:33.260023 22188 consensus_queue.cc:1035] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:33.303226 21149 raft_consensus.cc:1261] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:33.305246 22250 consensus_queue.cc:1035] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:33.305505 22185 raft_consensus.cc:2932] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
W20260812 19:12:33.313313 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 64734.9us
I20260812 19:12:33.314700 21320 raft_consensus.cc:1261] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:33.316802 22056 raft_consensus.cc:1261] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:33.321540 21320 raft_consensus.cc:2932] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.323122 22056 raft_consensus.cc:2932] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.331292 22138 consensus_queue.cc:1035] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:33.351181 22247 consensus_queue.cc:1035] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:33.355125 21035 catalog_manager.cc:5495] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.357715 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3f11f84389864fddb8350b0f94829938 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:33.367161 22188 raft_consensus.cc:2932] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.382020 22053 raft_consensus.cc:2932] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.387866 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f11f84389864fddb8350b0f94829938 with delete_type TABLET_DATA_TOMBSTONED (TS f741102bfe1441d1b453e5a838649059 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54786
I20260812 19:12:33.389424 21150 raft_consensus.cc:2932] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.378248 21322 raft_consensus.cc:2932] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.407989 21035 catalog_manager.cc:5495] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.416510 22255 tablet_replica.cc:330] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059: stopping tablet replica
I20260812 19:12:33.417824 22255 raft_consensus.cc:2229] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:33.419010 22255 raft_consensus.cc:2258] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:33.430996 22172 raft_consensus.cc:1050] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
W20260812 19:12:33.431087 22250 consensus_queue.cc:1159] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "93578836968649f6bfcaecc709871412" responder_term: 1 status { last_received { term: 1 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 1 index: 3 } }
I20260812 19:12:33.432437 21235 consensus_queue.cc:237] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:33.433719 22172 consensus_queue.cc:237] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:33.441949 22053 raft_consensus.cc:1261] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:33.440218 21322 raft_consensus.cc:1261] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:33.444959 22138 consensus_queue.cc:1035] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:12:33.449209 22138 consensus_queue.cc:1035] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:12:33.451073 22183 raft_consensus.cc:1050] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:33.457264 22255 ts_tablet_manager.cc:1905] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:33.457540 22188 raft_consensus.cc:2932] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.462137 21322 raft_consensus.cc:2932] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.454164 22183 consensus_queue.cc:237] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:33.468199 22053 raft_consensus.cc:2932] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.506438 21035 catalog_manager.cc:5495] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.508307 22056 raft_consensus.cc:1261] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:33.515357 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ae9be4809de24d32bdd888c6f9511d0f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:33.519279 22258 consensus_queue.cc:1035] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:33.527901 22055 raft_consensus.cc:1261] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:33.524982 21322 raft_consensus.cc:1261] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:33.528023 21319 raft_consensus.cc:1261] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:33.531865 21233 raft_consensus.cc:1261] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:33.534013 21233 raft_consensus.cc:1261] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:33.535558 22260 consensus_queue.cc:1035] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:33.542709 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet ae9be4809de24d32bdd888c6f9511d0f with delete_type TABLET_DATA_TOMBSTONED (TS f741102bfe1441d1b453e5a838649059 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54786
I20260812 19:12:33.549839 22260 consensus_queue.cc:1035] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:33.551069 22262 consensus_queue.cc:1035] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:33.564517 22255 ts_tablet_manager.cc:1918] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:33.565081 22255 log.cc:1196] T 3f11f84389864fddb8350b0f94829938 P f741102bfe1441d1b453e5a838649059: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-0-root/wals/3f11f84389864fddb8350b0f94829938
I20260812 19:12:33.565940 22258 consensus_queue.cc:1035] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:33.567150 22255 tablet_replica.cc:330] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059: stopping tablet replica
I20260812 19:12:33.568395 22255 raft_consensus.cc:2229] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:33.572122 21022 catalog_manager.cc:4841] TS f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): tablet 3f11f84389864fddb8350b0f94829938 (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:33.568152 22258 consensus_queue.cc:1035] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:12:33.582676 22255 raft_consensus.cc:2258] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:33.588461 22255 ts_tablet_manager.cc:1905] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:33.601424 22258 raft_consensus.cc:2932] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.613051 21235 raft_consensus.cc:2932] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.639256 22262 raft_consensus.cc:2932] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.645028 21235 raft_consensus.cc:2932] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.655592 21037 catalog_manager.cc:5495] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.687803 21037 catalog_manager.cc:5495] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.690332 21322 raft_consensus.cc:2932] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.692037 21320 raft_consensus.cc:2932] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.702028 22055 raft_consensus.cc:2932] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.706897 22255 ts_tablet_manager.cc:1918] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:33.709893 22255 log.cc:1196] T ae9be4809de24d32bdd888c6f9511d0f P f741102bfe1441d1b453e5a838649059: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-0-root/wals/ae9be4809de24d32bdd888c6f9511d0f
I20260812 19:12:33.712692 22056 raft_consensus.cc:2932] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.713968 21022 catalog_manager.cc:4841] TS f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): tablet ae9be4809de24d32bdd888c6f9511d0f (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:33.746668 21151 consensus_queue.cc:237] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:33.751488 22055 raft_consensus.cc:1261] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:33.752082 21235 raft_consensus.cc:1261] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:33.753396 22274 consensus_queue.cc:1035] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:12:33.755450 22262 consensus_queue.cc:1035] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:33.763060 22172 raft_consensus.cc:2932] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.775846 22055 raft_consensus.cc:2932] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.776500 21150 consensus_queue.cc:237] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:33.779309 21235 raft_consensus.cc:2932] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.782474 21233 raft_consensus.cc:1261] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:33.787235 22262 consensus_queue.cc:1035] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:33.794587 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet aff4931ad1bb4657aa3650a15b43ddcd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:33.795332 21035 catalog_manager.cc:5495] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.799903 22055 raft_consensus.cc:1261] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:33.799572 22274 raft_consensus.cc:2932] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.803635 22260 consensus_queue.cc:1035] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:33.810827 21316 tablet_service.cc:1514] Processing DeleteTablet for tablet aff4931ad1bb4657aa3650a15b43ddcd with delete_type TABLET_DATA_TOMBSTONED (TS 93578836968649f6bfcaecc709871412 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51536
I20260812 19:12:33.811971 22055 raft_consensus.cc:2932] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.824095 21233 raft_consensus.cc:2932] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.843052 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d6521960ce6040af919a9e227d2211ec with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:33.845786 22276 tablet_replica.cc:330] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412: stopping tablet replica
I20260812 19:12:33.850391 22276 raft_consensus.cc:2229] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:33.852649 22276 raft_consensus.cc:2258] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:33.853524 21036 catalog_manager.cc:5495] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:33.867594 21316 tablet_service.cc:1514] Processing DeleteTablet for tablet d6521960ce6040af919a9e227d2211ec with delete_type TABLET_DATA_TOMBSTONED (TS 93578836968649f6bfcaecc709871412 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51536
I20260812 19:12:33.874488 22276 ts_tablet_manager.cc:1905] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:33.891119 22276 ts_tablet_manager.cc:1918] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:33.891551 22276 log.cc:1196] T aff4931ad1bb4657aa3650a15b43ddcd P 93578836968649f6bfcaecc709871412: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-2-root/wals/aff4931ad1bb4657aa3650a15b43ddcd
I20260812 19:12:33.893118 21019 catalog_manager.cc:4841] TS 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): tablet aff4931ad1bb4657aa3650a15b43ddcd (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:33.893350 22276 tablet_replica.cc:330] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412: stopping tablet replica
I20260812 19:12:33.894485 22276 raft_consensus.cc:2229] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:33.895301 22276 raft_consensus.cc:2258] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:33.898523 22276 ts_tablet_manager.cc:1905] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:33.914577 22276 ts_tablet_manager.cc:1918] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:33.915011 22276 log.cc:1196] T d6521960ce6040af919a9e227d2211ec P 93578836968649f6bfcaecc709871412: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-2-root/wals/d6521960ce6040af919a9e227d2211ec
I20260812 19:12:33.916824 21019 catalog_manager.cc:4841] TS 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): tablet d6521960ce6040af919a9e227d2211ec (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:34.918710 21322 consensus_queue.cc:237] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:34.925834 21151 raft_consensus.cc:1261] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:34.925967 21235 raft_consensus.cc:1261] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:34.927817 22137 consensus_queue.cc:1035] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:34.928867 22275 consensus_queue.cc:1035] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260812 19:12:34.931489 21288 consensus_peers.cc:487] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 784d71ae37454b3a9e8023f268428468. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:34.938278 22137 raft_consensus.cc:2932] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:34.940075 21151 raft_consensus.cc:2932] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:34.940866 21235 raft_consensus.cc:2932] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:34.955322 21233 consensus_queue.cc:237] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:34.957527 21036 catalog_manager.cc:5495] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:34.974962 21322 raft_consensus.cc:1261] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 19:12:34.976301 21204 consensus_peers.cc:487] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc50dccc6345495aaec42b51dc30d299. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:34.982640 21151 raft_consensus.cc:1261] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:34.984349 22277 consensus_queue.cc:1035] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:34.995056 22277 raft_consensus.cc:2932] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:35.001031 21150 raft_consensus.cc:2932] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:35.017436 22247 consensus_queue.cc:1035] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:35.025440 21320 consensus_queue.cc:237] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:35.030232 21037 catalog_manager.cc:5495] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:35.048460 21322 raft_consensus.cc:2932] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
W20260812 19:12:35.052723 21288 consensus_peers.cc:487] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d58350d75f2f4a61b79148bd023ff99c. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:35.053454 21235 raft_consensus.cc:1261] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:35.055996 21150 raft_consensus.cc:1261] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:35.057577 22234 consensus_queue.cc:1035] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:35.060571 22275 consensus_queue.cc:1035] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:35.079564 22234 raft_consensus.cc:2932] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:35.081446 21151 raft_consensus.cc:2932] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:35.089744 21239 raft_consensus.cc:2932] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:35.104966 21239 consensus_queue.cc:237] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:35.105083 21035 catalog_manager.cc:5495] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260812 19:12:35.121941 21204 consensus_peers.cc:487] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc64ef202b904bcf809760ddfcd1f140. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:35.123319 21322 raft_consensus.cc:1261] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:35.124821 22247 consensus_queue.cc:1035] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:35.126945 21151 raft_consensus.cc:1261] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:35.128433 22250 consensus_queue.cc:1035] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:35.140494 22247 raft_consensus.cc:2932] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:35.142395 21322 raft_consensus.cc:2932] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:35.146257 21150 raft_consensus.cc:2932] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:35.162259 21036 catalog_manager.cc:5495] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:35.390450 22298 ts_tablet_manager.cc:927] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:35.394155 22298 tablet_copy_client.cc:316] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.130:33807
I20260812 19:12:35.395871 21255 tablet_copy_service.cc:140] P d1e1da0b9585460e8c365bf55679d4fa: Received BeginTabletCopySession request for tablet fc50dccc6345495aaec42b51dc30d299 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:57682)
I20260812 19:12:35.396438 21255 tablet_copy_service.cc:161] P d1e1da0b9585460e8c365bf55679d4fa: Beginning new tablet copy session on tablet fc50dccc6345495aaec42b51dc30d299 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:57682: session id = 63db10063a7d4eb2a9cc9dc952654706-fc50dccc6345495aaec42b51dc30d299
I20260812 19:12:35.405364 21255 tablet_copy_source_session.cc:205] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:35.408614 22298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc50dccc6345495aaec42b51dc30d299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:35.428670 22298 tablet_copy_client.cc:791] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:35.429289 22298 tablet_copy_client.cc:655] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:35.435281 22298 tablet_copy_client.cc:531] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:35.455789 22300 ts_tablet_manager.cc:927] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:35.456835 22298 tablet_bootstrap.cc:492] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:35.462693 22300 tablet_copy_client.cc:316] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.131:33285
I20260812 19:12:35.466409 21342 tablet_copy_service.cc:140] P 93578836968649f6bfcaecc709871412: Received BeginTabletCopySession request for tablet 784d71ae37454b3a9e8023f268428468 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:50042)
I20260812 19:12:35.467056 21342 tablet_copy_service.cc:161] P 93578836968649f6bfcaecc709871412: Beginning new tablet copy session on tablet 784d71ae37454b3a9e8023f268428468 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:50042: session id = 63db10063a7d4eb2a9cc9dc952654706-784d71ae37454b3a9e8023f268428468
I20260812 19:12:35.475203 21342 tablet_copy_source_session.cc:205] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:35.478838 22300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784d71ae37454b3a9e8023f268428468. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:35.482167 22298 tablet_bootstrap.cc:492] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:35.483148 22298 tablet_bootstrap.cc:492] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:35.483841 22298 ts_tablet_manager.cc:1397] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.015s
I20260812 19:12:35.486861 22298 raft_consensus.cc:348] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:35.487841 22298 raft_consensus.cc:729] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:35.488504 22298 consensus_queue.cc:260] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:35.492959 22298 ts_tablet_manager.cc:1428] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20260812 19:12:35.495463 21255 tablet_copy_service.cc:342] P d1e1da0b9585460e8c365bf55679d4fa: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-fc50dccc6345495aaec42b51dc30d299 received from {username='slave'} at 127.0.0.1:57682
I20260812 19:12:35.495863 21255 tablet_copy_service.cc:433] P d1e1da0b9585460e8c365bf55679d4fa: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-fc50dccc6345495aaec42b51dc30d299 on tablet fc50dccc6345495aaec42b51dc30d299 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:35.501271 22300 tablet_copy_client.cc:791] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:35.501979 22300 tablet_copy_client.cc:655] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:35.509593 22300 tablet_copy_client.cc:531] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:35.518102 22300 tablet_bootstrap.cc:492] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:35.538203 22298 ts_tablet_manager.cc:927] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:35.550572 22298 tablet_copy_client.cc:316] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.130:33807
I20260812 19:12:35.552444 21255 tablet_copy_service.cc:140] P d1e1da0b9585460e8c365bf55679d4fa: Received BeginTabletCopySession request for tablet dc64ef202b904bcf809760ddfcd1f140 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:57682)
I20260812 19:12:35.553073 21255 tablet_copy_service.cc:161] P d1e1da0b9585460e8c365bf55679d4fa: Beginning new tablet copy session on tablet dc64ef202b904bcf809760ddfcd1f140 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:57682: session id = 63db10063a7d4eb2a9cc9dc952654706-dc64ef202b904bcf809760ddfcd1f140
I20260812 19:12:35.564185 22300 tablet_bootstrap.cc:492] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:35.565747 22300 tablet_bootstrap.cc:492] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:35.566756 22300 ts_tablet_manager.cc:1397] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.049s user 0.024s sys 0.015s
I20260812 19:12:35.571175 22300 raft_consensus.cc:348] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:35.572345 22300 raft_consensus.cc:729] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:35.573307 22300 consensus_queue.cc:260] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:35.575600 21255 tablet_copy_source_session.cc:205] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:35.579368 22300 ts_tablet_manager.cc:1428] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260812 19:12:35.584551 22298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc64ef202b904bcf809760ddfcd1f140. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:35.599231 21342 tablet_copy_service.cc:342] P 93578836968649f6bfcaecc709871412: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-784d71ae37454b3a9e8023f268428468 received from {username='slave'} at 127.0.0.1:50042
I20260812 19:12:35.599869 21342 tablet_copy_service.cc:433] P 93578836968649f6bfcaecc709871412: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-784d71ae37454b3a9e8023f268428468 on tablet 784d71ae37454b3a9e8023f268428468 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:35.611987 22300 ts_tablet_manager.cc:927] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:35.613749 22298 tablet_copy_client.cc:791] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:35.614661 22298 tablet_copy_client.cc:655] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:35.615250 22300 tablet_copy_client.cc:316] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.131:33285
I20260812 19:12:35.618170 21342 tablet_copy_service.cc:140] P 93578836968649f6bfcaecc709871412: Received BeginTabletCopySession request for tablet d58350d75f2f4a61b79148bd023ff99c from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:50042)
I20260812 19:12:35.618822 21342 tablet_copy_service.cc:161] P 93578836968649f6bfcaecc709871412: Beginning new tablet copy session on tablet d58350d75f2f4a61b79148bd023ff99c from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:50042: session id = 63db10063a7d4eb2a9cc9dc952654706-d58350d75f2f4a61b79148bd023ff99c
I20260812 19:12:35.620975 22298 tablet_copy_client.cc:531] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:35.628607 21342 tablet_copy_source_session.cc:205] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:35.632462 22298 tablet_bootstrap.cc:492] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:35.632671 22300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58350d75f2f4a61b79148bd023ff99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:35.656080 22300 tablet_copy_client.cc:791] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:35.656733 22300 tablet_copy_client.cc:655] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:35.656805 22298 tablet_bootstrap.cc:492] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:35.657828 22298 tablet_bootstrap.cc:492] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:35.658563 22298 ts_tablet_manager.cc:1397] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.012s
I20260812 19:12:35.661594 22298 raft_consensus.cc:348] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:35.662539 22298 raft_consensus.cc:729] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:35.663285 22298 consensus_queue.cc:260] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:35.664557 22300 tablet_copy_client.cc:531] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:35.665922 22298 ts_tablet_manager.cc:1428] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260812 19:12:35.668025 21255 tablet_copy_service.cc:342] P d1e1da0b9585460e8c365bf55679d4fa: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-dc64ef202b904bcf809760ddfcd1f140 received from {username='slave'} at 127.0.0.1:57682
I20260812 19:12:35.668591 21255 tablet_copy_service.cc:433] P d1e1da0b9585460e8c365bf55679d4fa: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-dc64ef202b904bcf809760ddfcd1f140 on tablet dc64ef202b904bcf809760ddfcd1f140 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:35.679847 22300 tablet_bootstrap.cc:492] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:35.701303 22300 tablet_bootstrap.cc:492] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:35.702384 22300 tablet_bootstrap.cc:492] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:35.703233 22300 ts_tablet_manager.cc:1397] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.005s
I20260812 19:12:35.706187 22300 raft_consensus.cc:348] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:35.707162 22300 raft_consensus.cc:729] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:35.707870 22300 consensus_queue.cc:260] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:35.710455 22300 ts_tablet_manager.cc:1428] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260812 19:12:35.721915 21342 tablet_copy_service.cc:342] P 93578836968649f6bfcaecc709871412: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-d58350d75f2f4a61b79148bd023ff99c received from {username='slave'} at 127.0.0.1:50042
I20260812 19:12:35.722306 21342 tablet_copy_service.cc:433] P 93578836968649f6bfcaecc709871412: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-d58350d75f2f4a61b79148bd023ff99c on tablet d58350d75f2f4a61b79148bd023ff99c with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:35.914222 22055 raft_consensus.cc:1203] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:35.983465 22055 raft_consensus.cc:1203] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:36.011502 22055 raft_consensus.cc:1203] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:36.085568 22055 raft_consensus.cc:1203] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:36.384719 22297 raft_consensus.cc:1050] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:36.387508 22297 consensus_queue.cc:237] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:36.402884 22055 raft_consensus.cc:1261] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:36.404871 22297 consensus_queue.cc:1035] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:36.405970 21150 raft_consensus.cc:1261] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:36.407352 21322 raft_consensus.cc:1261] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:36.407732 22284 consensus_queue.cc:1035] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:36.409824 22284 consensus_queue.cc:1035] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:12:36.431193 21322 raft_consensus.cc:2932] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.431344 22311 raft_consensus.cc:2932] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.446568 22055 raft_consensus.cc:2932] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.476832 21151 raft_consensus.cc:2932] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.483696 21035 catalog_manager.cc:5495] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.502683 22137 raft_consensus.cc:1050] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:36.506084 22137 consensus_queue.cc:237] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:36.514670 22055 raft_consensus.cc:1261] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:36.517009 21150 raft_consensus.cc:1261] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:36.521556 21235 raft_consensus.cc:1261] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:36.521097 21233 consensus_queue.cc:237] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:36.518055 22137 consensus_queue.cc:1035] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260812 19:12:36.525701 22306 consensus_queue.cc:1035] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:12:36.530779 21322 raft_consensus.cc:1261] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:36.530781 22055 raft_consensus.cc:1261] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:36.532984 22317 consensus_queue.cc:1035] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:36.534965 22315 consensus_queue.cc:1035] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:12:36.551112 22317 raft_consensus.cc:2932] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.537869 22324 consensus_queue.cc:1035] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260812 19:12:36.554646 22055 raft_consensus.cc:2932] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.576098 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet fc50dccc6345495aaec42b51dc30d299 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:36.577131 21034 catalog_manager.cc:5495] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:36.567176 21322 raft_consensus.cc:2932] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.593937 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet fc50dccc6345495aaec42b51dc30d299 with delete_type TABLET_DATA_TOMBSTONED (TS f741102bfe1441d1b453e5a838649059 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54786
I20260812 19:12:36.604307 22331 tablet_replica.cc:330] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059: stopping tablet replica
I20260812 19:12:36.625339 22275 raft_consensus.cc:2932] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.628027 22284 raft_consensus.cc:1050] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:36.631348 22284 consensus_queue.cc:237] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:36.640244 22331 raft_consensus.cc:2229] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:36.641182 22331 raft_consensus.cc:2258] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:36.634164 21239 raft_consensus.cc:2932] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.645900 21150 raft_consensus.cc:2932] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.652626 22331 ts_tablet_manager.cc:1905] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:36.660674 22055 raft_consensus.cc:2932] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.663681 22306 raft_consensus.cc:1050] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:36.670054 22306 consensus_queue.cc:237] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:36.677821 21037 catalog_manager.cc:5495] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.690527 22053 raft_consensus.cc:1261] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:36.692795 22306 consensus_queue.cc:1035] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:36.697394 21151 raft_consensus.cc:1261] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:36.693209 21239 raft_consensus.cc:1261] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:36.703143 21322 consensus_queue.cc:237] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:36.700577 22306 consensus_queue.cc:1035] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260812 19:12:36.706121 22275 consensus_queue.cc:1035] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:12:36.724959 21150 raft_consensus.cc:1261] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:36.726886 22056 raft_consensus.cc:1261] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:36.729202 22275 consensus_queue.cc:1035] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:12:36.729663 22317 consensus_queue.cc:1035] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:36.732067 22053 raft_consensus.cc:1261] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:36.738332 21322 raft_consensus.cc:1261] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:36.752781 22306 consensus_queue.cc:1035] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260812 19:12:36.777273 22317 consensus_queue.cc:1035] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260812 19:12:36.785140 22275 raft_consensus.cc:2932] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.802284 22326 raft_consensus.cc:2932] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.804585 22055 raft_consensus.cc:2932] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.808878 22053 raft_consensus.cc:2932] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
W20260812 19:12:36.821784 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 82738.4us
W20260812 19:12:36.841542 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 102485us
I20260812 19:12:36.852166 21151 raft_consensus.cc:1261] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:36.856760 21019 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 784d71ae37454b3a9e8023f268428468 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:36.873384 22331 ts_tablet_manager.cc:1918] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:36.873951 22331 log.cc:1196] T fc50dccc6345495aaec42b51dc30d299 P f741102bfe1441d1b453e5a838649059: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-0-root/wals/fc50dccc6345495aaec42b51dc30d299
I20260812 19:12:36.878527 21022 catalog_manager.cc:4841] TS f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): tablet fc50dccc6345495aaec42b51dc30d299 (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:36.859735 21150 raft_consensus.cc:2932] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.869638 21157 raft_consensus.cc:2932] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.883958 21037 catalog_manager.cc:5495] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260812 19:12:36.891337 22338 log.cc:924] Time spent T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.073s user 0.000s sys 0.002s
W20260812 19:12:36.894011 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 58352.6us
I20260812 19:12:36.895965 21034 catalog_manager.cc:5495] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.899101 22311 consensus_queue.cc:1035] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:36.908375 21239 raft_consensus.cc:2932] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.920392 22316 raft_consensus.cc:2932] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.934026 21328 raft_consensus.cc:2932] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.938378 21322 consensus_queue.cc:237] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:36.950049 21219 tablet_service.cc:1514] Processing DeleteTablet for tablet 784d71ae37454b3a9e8023f268428468 with delete_type TABLET_DATA_TOMBSTONED (TS d1e1da0b9585460e8c365bf55679d4fa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37890
I20260812 19:12:36.975248 22053 raft_consensus.cc:2932] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:36.984652 22055 raft_consensus.cc:1261] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:36.986748 22137 consensus_queue.cc:1035] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:37.000862 22326 raft_consensus.cc:2932] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:37.011842 21151 raft_consensus.cc:1261] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:37.014074 22055 raft_consensus.cc:2932] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:37.042702 22344 consensus_queue.cc:1035] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:37.049664 21150 raft_consensus.cc:2932] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:37.061056 21037 catalog_manager.cc:5495] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:37.085978 22350 tablet_replica.cc:330] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa: stopping tablet replica
I20260812 19:12:37.087291 22350 raft_consensus.cc:2229] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:37.088089 22350 raft_consensus.cc:2258] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:37.089469 21019 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d58350d75f2f4a61b79148bd023ff99c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:37.094583 22350 ts_tablet_manager.cc:1905] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:37.100641 21150 raft_consensus.cc:2932] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:37.105614 21034 catalog_manager.cc:5495] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 reported cstate change: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:37.144202 22350 ts_tablet_manager.cc:1918] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:37.144724 22350 log.cc:1196] T 784d71ae37454b3a9e8023f268428468 P d1e1da0b9585460e8c365bf55679d4fa: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-1-root/wals/784d71ae37454b3a9e8023f268428468
I20260812 19:12:37.150163 21219 tablet_service.cc:1514] Processing DeleteTablet for tablet d58350d75f2f4a61b79148bd023ff99c with delete_type TABLET_DATA_TOMBSTONED (TS d1e1da0b9585460e8c365bf55679d4fa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37890
I20260812 19:12:37.152175 21022 catalog_manager.cc:4841] TS d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): tablet 784d71ae37454b3a9e8023f268428468 (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:37.154435 22350 tablet_replica.cc:330] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa: stopping tablet replica
I20260812 19:12:37.167254 22350 raft_consensus.cc:2229] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:37.168139 22350 raft_consensus.cc:2258] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:37.172585 22350 ts_tablet_manager.cc:1905] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:37.174512 21239 consensus_queue.cc:237] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:37.183248 22055 raft_consensus.cc:1261] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:37.183502 21150 raft_consensus.cc:1261] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:37.189132 22317 consensus_queue.cc:1035] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:37.191187 22350 ts_tablet_manager.cc:1918] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:37.191672 22350 log.cc:1196] T d58350d75f2f4a61b79148bd023ff99c P d1e1da0b9585460e8c365bf55679d4fa: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-1-root/wals/d58350d75f2f4a61b79148bd023ff99c
I20260812 19:12:37.193759 21022 catalog_manager.cc:4841] TS d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): tablet d58350d75f2f4a61b79148bd023ff99c (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:37.194515 22297 consensus_queue.cc:1035] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:12:37.204825 22317 raft_consensus.cc:2932] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:37.207147 22055 raft_consensus.cc:2932] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:37.209841 21149 raft_consensus.cc:2932] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:37.218166 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet dc64ef202b904bcf809760ddfcd1f140 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:37.230226 21035 catalog_manager.cc:5495] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:37.256671 21316 tablet_service.cc:1514] Processing DeleteTablet for tablet dc64ef202b904bcf809760ddfcd1f140 with delete_type TABLET_DATA_TOMBSTONED (TS 93578836968649f6bfcaecc709871412 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51536
I20260812 19:12:37.262603 22352 tablet_replica.cc:330] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412: stopping tablet replica
I20260812 19:12:37.263895 22352 raft_consensus.cc:2229] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:37.264750 22352 raft_consensus.cc:2258] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:37.268622 22352 ts_tablet_manager.cc:1905] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:37.283366 22352 ts_tablet_manager.cc:1918] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:37.283818 22352 log.cc:1196] T dc64ef202b904bcf809760ddfcd1f140 P 93578836968649f6bfcaecc709871412: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-2-root/wals/dc64ef202b904bcf809760ddfcd1f140
I20260812 19:12:37.285244 21019 catalog_manager.cc:4841] TS 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): tablet dc64ef202b904bcf809760ddfcd1f140 (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:38.300602 21151 consensus_queue.cc:237] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:38.306735 21319 raft_consensus.cc:1261] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 19:12:38.308634 21115 consensus_peers.cc:487] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7825e8ab0b2e4d7abbb7dd19c9e3ad69. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:38.309414 21235 raft_consensus.cc:1261] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:38.309878 22353 consensus_queue.cc:1035] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:38.326467 22347 consensus_queue.cc:1035] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:38.338395 22335 raft_consensus.cc:2932] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:38.340605 21319 raft_consensus.cc:2932] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:38.347505 21233 raft_consensus.cc:2932] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:38.360510 21035 catalog_manager.cc:5495] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:38.379640 21151 consensus_queue.cc:237] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:38.393919 21319 raft_consensus.cc:1261] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 19:12:38.394816 21115 consensus_peers.cc:487] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41fc49d178354e629e97311498b381ec. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:38.395828 21235 raft_consensus.cc:1261] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:38.396219 22335 consensus_queue.cc:1035] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:38.397919 22353 consensus_queue.cc:1035] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:38.411000 22335 raft_consensus.cc:2932] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:38.413903 21319 raft_consensus.cc:2932] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:38.415627 21235 raft_consensus.cc:2932] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:38.435935 21233 consensus_queue.cc:237] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:38.436563 21037 catalog_manager.cc:5495] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
W20260812 19:12:38.461064 21204 consensus_peers.cc:487] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4580f6a22878438bb77f8d43fbddf459. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:38.463938 21319 raft_consensus.cc:1261] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:38.465740 22356 consensus_queue.cc:1035] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:38.470369 21151 raft_consensus.cc:1261] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:38.476331 22297 consensus_queue.cc:1035] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:38.487516 22356 raft_consensus.cc:2932] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:38.495893 21319 raft_consensus.cc:2932] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:38.513717 21150 raft_consensus.cc:2932] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:38.524605 21037 catalog_manager.cc:5495] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:38.539125 21233 consensus_queue.cc:237] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
W20260812 19:12:38.572392 21204 consensus_peers.cc:487] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6309c277c104b23a58d4824d764d397. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:38.578919 21319 raft_consensus.cc:1261] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:38.581461 22356 consensus_queue.cc:1035] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:38.583299 21150 raft_consensus.cc:1261] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:38.595340 22356 consensus_queue.cc:1035] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:38.597559 22316 raft_consensus.cc:2932] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:38.608882 21322 raft_consensus.cc:2932] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:38.611044 21150 raft_consensus.cc:2932] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:38.628163 21037 catalog_manager.cc:5495] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:38.816439 22378 ts_tablet_manager.cc:927] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551)
I20260812 19:12:38.818519 22378 tablet_copy_client.cc:316] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.129:36551
I20260812 19:12:38.820268 21169 tablet_copy_service.cc:140] P f741102bfe1441d1b453e5a838649059: Received BeginTabletCopySession request for tablet 7825e8ab0b2e4d7abbb7dd19c9e3ad69 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:51004)
I20260812 19:12:38.820796 21169 tablet_copy_service.cc:161] P f741102bfe1441d1b453e5a838649059: Beginning new tablet copy session on tablet 7825e8ab0b2e4d7abbb7dd19c9e3ad69 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:51004: session id = 63db10063a7d4eb2a9cc9dc952654706-7825e8ab0b2e4d7abbb7dd19c9e3ad69
I20260812 19:12:38.828441 21169 tablet_copy_source_session.cc:205] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:38.831733 22378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7825e8ab0b2e4d7abbb7dd19c9e3ad69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:38.838732 22380 ts_tablet_manager.cc:927] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551)
I20260812 19:12:38.847450 22380 tablet_copy_client.cc:316] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.129:36551
I20260812 19:12:38.849459 21169 tablet_copy_service.cc:140] P f741102bfe1441d1b453e5a838649059: Received BeginTabletCopySession request for tablet 41fc49d178354e629e97311498b381ec from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:51004)
I20260812 19:12:38.850064 21169 tablet_copy_service.cc:161] P f741102bfe1441d1b453e5a838649059: Beginning new tablet copy session on tablet 41fc49d178354e629e97311498b381ec from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:51004: session id = 63db10063a7d4eb2a9cc9dc952654706-41fc49d178354e629e97311498b381ec
I20260812 19:12:38.853112 22378 tablet_copy_client.cc:791] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:38.853806 22378 tablet_copy_client.cc:655] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:38.858809 21169 tablet_copy_source_session.cc:205] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:38.862002 22380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41fc49d178354e629e97311498b381ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:38.864696 22378 tablet_copy_client.cc:531] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:38.877449 22378 tablet_bootstrap.cc:492] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:38.894273 22380 tablet_copy_client.cc:791] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:38.895058 22380 tablet_copy_client.cc:655] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:38.900691 22380 tablet_copy_client.cc:531] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:38.912204 22378 tablet_bootstrap.cc:492] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:38.913383 22378 tablet_bootstrap.cc:492] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:38.913945 22380 tablet_bootstrap.cc:492] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:38.914578 22378 ts_tablet_manager.cc:1397] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.037s user 0.024s sys 0.004s
I20260812 19:12:38.918499 22378 raft_consensus.cc:348] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:38.919409 22378 raft_consensus.cc:729] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:38.920154 22378 consensus_queue.cc:260] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:38.923722 22378 ts_tablet_manager.cc:1428] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260812 19:12:38.925804 21169 tablet_copy_service.cc:342] P f741102bfe1441d1b453e5a838649059: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-7825e8ab0b2e4d7abbb7dd19c9e3ad69 received from {username='slave'} at 127.0.0.1:51004
I20260812 19:12:38.926267 21169 tablet_copy_service.cc:433] P f741102bfe1441d1b453e5a838649059: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-7825e8ab0b2e4d7abbb7dd19c9e3ad69 on tablet 7825e8ab0b2e4d7abbb7dd19c9e3ad69 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:38.955354 22380 tablet_bootstrap.cc:492] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:38.956858 22380 tablet_bootstrap.cc:492] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:38.957719 22380 ts_tablet_manager.cc:1397] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.044s user 0.029s sys 0.013s
I20260812 19:12:38.961731 22380 raft_consensus.cc:348] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:38.962716 22380 raft_consensus.cc:729] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:38.963459 22380 consensus_queue.cc:260] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:38.966003 22380 ts_tablet_manager.cc:1428] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260812 19:12:38.968226 21169 tablet_copy_service.cc:342] P f741102bfe1441d1b453e5a838649059: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-41fc49d178354e629e97311498b381ec received from {username='slave'} at 127.0.0.1:51004
I20260812 19:12:38.968662 21169 tablet_copy_service.cc:433] P f741102bfe1441d1b453e5a838649059: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-41fc49d178354e629e97311498b381ec on tablet 41fc49d178354e629e97311498b381ec with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:38.987528 22380 ts_tablet_manager.cc:927] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:38.993954 22380 tablet_copy_client.cc:316] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.130:33807
I20260812 19:12:38.996989 21255 tablet_copy_service.cc:140] P d1e1da0b9585460e8c365bf55679d4fa: Received BeginTabletCopySession request for tablet 4580f6a22878438bb77f8d43fbddf459 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:57682)
I20260812 19:12:38.997568 21255 tablet_copy_service.cc:161] P d1e1da0b9585460e8c365bf55679d4fa: Beginning new tablet copy session on tablet 4580f6a22878438bb77f8d43fbddf459 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:57682: session id = 63db10063a7d4eb2a9cc9dc952654706-4580f6a22878438bb77f8d43fbddf459
I20260812 19:12:39.006371 21255 tablet_copy_source_session.cc:205] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:39.069818 22378 ts_tablet_manager.cc:927] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:39.090770 22380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4580f6a22878438bb77f8d43fbddf459. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:39.103202 22378 tablet_copy_client.cc:316] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.130:33807
I20260812 19:12:39.154402 21255 tablet_copy_service.cc:140] P d1e1da0b9585460e8c365bf55679d4fa: Received BeginTabletCopySession request for tablet c6309c277c104b23a58d4824d764d397 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:57682)
I20260812 19:12:39.155242 21255 tablet_copy_service.cc:161] P d1e1da0b9585460e8c365bf55679d4fa: Beginning new tablet copy session on tablet c6309c277c104b23a58d4824d764d397 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:57682: session id = 63db10063a7d4eb2a9cc9dc952654706-c6309c277c104b23a58d4824d764d397
I20260812 19:12:39.176062 21255 tablet_copy_source_session.cc:205] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:39.195322 22380 tablet_copy_client.cc:791] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:39.196565 22380 tablet_copy_client.cc:655] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:39.217800 22380 tablet_copy_client.cc:531] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260812 19:12:39.250048 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 58952.5us
I20260812 19:12:39.258241 22378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6309c277c104b23a58d4824d764d397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:39.294584 22380 tablet_bootstrap.cc:492] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:39.354182 22380 tablet_bootstrap.cc:492] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:39.355453 22380 tablet_bootstrap.cc:492] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:39.356556 22380 ts_tablet_manager.cc:1397] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.062s user 0.020s sys 0.012s
I20260812 19:12:39.361264 22378 tablet_copy_client.cc:791] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:39.362169 22378 tablet_copy_client.cc:655] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:39.361558 22380 raft_consensus.cc:348] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:39.363576 22380 raft_consensus.cc:729] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:39.364666 22380 consensus_queue.cc:260] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:39.371534 22380 ts_tablet_manager.cc:1428] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.014s user 0.009s sys 0.001s
I20260812 19:12:39.446455 22378 tablet_copy_client.cc:531] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:39.457019 22378 tablet_bootstrap.cc:492] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:39.466387 21254 tablet_copy_service.cc:342] P d1e1da0b9585460e8c365bf55679d4fa: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-4580f6a22878438bb77f8d43fbddf459 received from {username='slave'} at 127.0.0.1:57682
I20260812 19:12:39.467036 21254 tablet_copy_service.cc:433] P d1e1da0b9585460e8c365bf55679d4fa: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-4580f6a22878438bb77f8d43fbddf459 on tablet 4580f6a22878438bb77f8d43fbddf459 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:39.484685 22378 tablet_bootstrap.cc:492] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:39.485939 22378 tablet_bootstrap.cc:492] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:39.486771 22378 ts_tablet_manager.cc:1397] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.003s
I20260812 19:12:39.490422 22378 raft_consensus.cc:348] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
W20260812 19:12:39.491029 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 184960us
I20260812 19:12:39.497918 22378 raft_consensus.cc:729] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:39.501745 22378 consensus_queue.cc:260] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:39.512163 22378 ts_tablet_manager.cc:1428] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.025s user 0.008s sys 0.000s
I20260812 19:12:39.525483 22055 raft_consensus.cc:1203] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20260812 19:12:39.595773 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 81932.1us
W20260812 19:12:39.659088 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 48707.1us
W20260812 19:12:39.707569 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 63744.3us
W20260812 19:12:39.715605 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 48603.7us
I20260812 19:12:39.527746 22056 raft_consensus.cc:1203] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:39.776381 21254 tablet_copy_service.cc:342] P d1e1da0b9585460e8c365bf55679d4fa: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-c6309c277c104b23a58d4824d764d397 received from {username='slave'} at 127.0.0.1:57682
I20260812 19:12:39.776860 21254 tablet_copy_service.cc:433] P d1e1da0b9585460e8c365bf55679d4fa: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-c6309c277c104b23a58d4824d764d397 on tablet c6309c277c104b23a58d4824d764d397 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:39.784428 22053 raft_consensus.cc:1203] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:39.788774 22055 raft_consensus.cc:1203] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:40.030447 22393 raft_consensus.cc:1050] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:40.034879 22393 consensus_queue.cc:237] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:40.054961 21328 raft_consensus.cc:1261] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:40.057444 22391 consensus_queue.cc:1035] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:40.061090 21241 raft_consensus.cc:1261] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:40.063117 22391 consensus_queue.cc:1035] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:40.065388 22053 raft_consensus.cc:1261] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:40.068660 22391 consensus_queue.cc:1035] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:12:40.088320 21327 raft_consensus.cc:2932] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.095682 22055 raft_consensus.cc:2932] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.086390 22393 raft_consensus.cc:2932] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.103221 22347 raft_consensus.cc:1050] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:40.105928 22347 consensus_queue.cc:237] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:40.123183 21035 catalog_manager.cc:5495] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.125814 21241 raft_consensus.cc:2932] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.147745 22053 raft_consensus.cc:1261] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:40.148416 21327 raft_consensus.cc:1261] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:40.149650 21244 raft_consensus.cc:1261] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:40.150097 22393 consensus_queue.cc:1035] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:40.159868 22410 raft_consensus.cc:1050] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:40.165566 22410 consensus_queue.cc:237] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:40.168850 22393 consensus_queue.cc:1035] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:40.173573 22372 raft_consensus.cc:1050] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:40.176743 22372 consensus_queue.cc:237] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:40.181727 22393 consensus_queue.cc:1035] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:40.211614 21327 raft_consensus.cc:1261] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:40.211601 22053 raft_consensus.cc:1261] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:40.215430 22401 consensus_queue.cc:1035] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:40.219022 22053 raft_consensus.cc:1261] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:40.221462 21153 raft_consensus.cc:1261] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:40.221665 22297 consensus_queue.cc:1035] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:40.225930 22389 consensus_queue.cc:1035] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:40.228920 21327 raft_consensus.cc:2932] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.231464 22356 consensus_queue.cc:1035] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:40.228605 22347 raft_consensus.cc:2932] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.271641 21152 raft_consensus.cc:1261] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:40.272322 21320 raft_consensus.cc:1261] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:40.270737 22055 raft_consensus.cc:2932] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.273949 22372 consensus_queue.cc:1035] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260812 19:12:40.281234 22389 consensus_queue.cc:1035] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:40.342144 21235 raft_consensus.cc:2932] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.343647 22401 raft_consensus.cc:2932] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.345019 22297 raft_consensus.cc:2932] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.348896 22056 raft_consensus.cc:2932] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.348875 22052 raft_consensus.cc:2932] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.352993 21153 raft_consensus.cc:2932] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.371816 21156 raft_consensus.cc:2932] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.390515 21320 raft_consensus.cc:2932] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.394783 21327 raft_consensus.cc:2932] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.410250 21034 catalog_manager.cc:5495] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.421452 21153 consensus_queue.cc:237] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:40.437081 21036 catalog_manager.cc:5495] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.441180 21036 catalog_manager.cc:5495] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.445991 21327 raft_consensus.cc:1261] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:40.449313 22056 raft_consensus.cc:1261] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:40.454480 22404 consensus_queue.cc:1035] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:40.455768 22347 consensus_queue.cc:1035] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:12:40.465332 22412 raft_consensus.cc:2932] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.470940 21320 raft_consensus.cc:2932] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.487840 21240 consensus_queue.cc:237] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:40.496184 21152 consensus_queue.cc:237] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:40.495841 22056 raft_consensus.cc:2932] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.496491 21235 consensus_queue.cc:237] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:40.502643 22053 raft_consensus.cc:1261] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:40.506033 21327 raft_consensus.cc:1261] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:40.506125 22053 raft_consensus.cc:1261] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:40.507748 22297 consensus_queue.cc:1035] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:40.509361 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 41fc49d178354e629e97311498b381ec with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:40.509701 22410 consensus_queue.cc:1035] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:12:40.509409 21235 raft_consensus.cc:1261] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:40.507951 22404 consensus_queue.cc:1035] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:40.522354 22401 raft_consensus.cc:2932] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.524287 21034 catalog_manager.cc:5495] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.528405 22412 consensus_queue.cc:1035] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:40.530680 22056 raft_consensus.cc:2932] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.536763 21152 raft_consensus.cc:1261] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:40.539512 22053 raft_consensus.cc:1261] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:40.539552 21320 raft_consensus.cc:2932] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.544092 22347 raft_consensus.cc:2932] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.540243 22410 consensus_queue.cc:1035] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260812 19:12:40.541500 21356 debug-util.cc:398] Leaking SignalData structure 0x7b08002050e0 after lost signal to thread 19742
W20260812 19:12:40.547824 21356 debug-util.cc:398] Leaking SignalData structure 0x7b08001347a0 after lost signal to thread 21077
W20260812 19:12:40.549078 21356 debug-util.cc:398] Leaking SignalData structure 0x7b08001e5980 after lost signal to thread 21185
W20260812 19:12:40.550244 21356 debug-util.cc:398] Leaking SignalData structure 0x7b08002df980 after lost signal to thread 21270
I20260812 19:12:40.551715 22356 consensus_queue.cc:1035] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260812 19:12:40.556252 21356 debug-util.cc:398] Leaking SignalData structure 0x7b080029ae00 after lost signal to thread 21359
I20260812 19:12:40.557313 21235 raft_consensus.cc:2932] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.562989 22356 raft_consensus.cc:2932] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.565418 21152 raft_consensus.cc:2932] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.565702 22053 raft_consensus.cc:2932] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.587527 21219 tablet_service.cc:1514] Processing DeleteTablet for tablet 41fc49d178354e629e97311498b381ec with delete_type TABLET_DATA_TOMBSTONED (TS d1e1da0b9585460e8c365bf55679d4fa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37890
W20260812 19:12:40.592484 21356 debug-util.cc:398] Leaking SignalData structure 0x7b0800160b40 after lost signal to thread 22084
W20260812 19:12:40.598985 21356 debug-util.cc:398] Leaking SignalData structure 0x7b080029f300 after lost signal to thread 22374
I20260812 19:12:40.606832 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4580f6a22878438bb77f8d43fbddf459 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:40.607604 21034 catalog_manager.cc:5495] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.611817 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 7825e8ab0b2e4d7abbb7dd19c9e3ad69 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:40.600033 22056 raft_consensus.cc:2932] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.636476 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c6309c277c104b23a58d4824d764d397 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:40.666895 22437 tablet_replica.cc:330] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa: stopping tablet replica
I20260812 19:12:40.669085 22437 raft_consensus.cc:2229] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:40.670145 22437 raft_consensus.cc:2258] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:40.672515 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet 4580f6a22878438bb77f8d43fbddf459 with delete_type TABLET_DATA_TOMBSTONED (TS f741102bfe1441d1b453e5a838649059 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54786
I20260812 19:12:40.685837 22437 ts_tablet_manager.cc:1905] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:40.692610 21037 catalog_manager.cc:5495] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.694710 22438 tablet_replica.cc:330] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059: stopping tablet replica
I20260812 19:12:40.698976 22438 raft_consensus.cc:2229] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:40.700968 22438 raft_consensus.cc:2258] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:40.699012 21037 catalog_manager.cc:5495] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:40.709254 22438 ts_tablet_manager.cc:1905] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:40.733860 21316 tablet_service.cc:1514] Processing DeleteTablet for tablet 7825e8ab0b2e4d7abbb7dd19c9e3ad69 with delete_type TABLET_DATA_TOMBSTONED (TS 93578836968649f6bfcaecc709871412 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51536
I20260812 19:12:40.734275 21315 tablet_service.cc:1514] Processing DeleteTablet for tablet c6309c277c104b23a58d4824d764d397 with delete_type TABLET_DATA_TOMBSTONED (TS 93578836968649f6bfcaecc709871412 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51536
I20260812 19:12:40.736457 22439 tablet_replica.cc:330] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412: stopping tablet replica
I20260812 19:12:40.739334 22439 raft_consensus.cc:2229] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:40.740406 22439 raft_consensus.cc:2258] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:40.749433 22438 ts_tablet_manager.cc:1918] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:40.750126 22438 log.cc:1196] T 4580f6a22878438bb77f8d43fbddf459 P f741102bfe1441d1b453e5a838649059: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-0-root/wals/4580f6a22878438bb77f8d43fbddf459
I20260812 19:12:40.752628 21022 catalog_manager.cc:4841] TS f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): tablet 4580f6a22878438bb77f8d43fbddf459 (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:40.754999 22437 ts_tablet_manager.cc:1918] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:40.755602 22437 log.cc:1196] T 41fc49d178354e629e97311498b381ec P d1e1da0b9585460e8c365bf55679d4fa: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-1-root/wals/41fc49d178354e629e97311498b381ec
I20260812 19:12:40.756398 22439 ts_tablet_manager.cc:1905] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:40.757799 21022 catalog_manager.cc:4841] TS d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): tablet 41fc49d178354e629e97311498b381ec (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:40.779237 22439 ts_tablet_manager.cc:1918] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:40.779882 22439 log.cc:1196] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 93578836968649f6bfcaecc709871412: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-2-root/wals/7825e8ab0b2e4d7abbb7dd19c9e3ad69
I20260812 19:12:40.781900 22439 tablet_replica.cc:330] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412: stopping tablet replica
I20260812 19:12:40.782300 21019 catalog_manager.cc:4841] TS 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): tablet 7825e8ab0b2e4d7abbb7dd19c9e3ad69 (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:40.783244 22439 raft_consensus.cc:2229] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:40.783973 22439 raft_consensus.cc:2258] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:40.788003 22439 ts_tablet_manager.cc:1905] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:40.803383 22439 ts_tablet_manager.cc:1918] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:40.803862 22439 log.cc:1196] T c6309c277c104b23a58d4824d764d397 P 93578836968649f6bfcaecc709871412: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-2-root/wals/c6309c277c104b23a58d4824d764d397
I20260812 19:12:40.805503 21019 catalog_manager.cc:4841] TS 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): tablet c6309c277c104b23a58d4824d764d397 (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:41.800148 21320 consensus_queue.cc:237] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:41.821798 21235 raft_consensus.cc:1261] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:41.823390 21156 raft_consensus.cc:1261] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:41.824218 22326 consensus_queue.cc:1035] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:41.825877 22442 consensus_queue.cc:1035] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260812 19:12:41.828666 21288 consensus_peers.cc:487] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6eaa29a6fe94976a6124729431bd8c8. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:41.845752 22326 raft_consensus.cc:2932] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:41.847798 21235 raft_consensus.cc:2932] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:41.855507 21156 raft_consensus.cc:2932] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:41.879093 21152 consensus_queue.cc:237] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:41.883960 21034 catalog_manager.cc:5495] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:41.905511 21327 raft_consensus.cc:1261] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:41.907487 22347 consensus_queue.cc:1035] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:41.919968 21235 raft_consensus.cc:1261] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 19:12:41.923589 21115 consensus_peers.cc:487] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfa25b5175f14002a8b5ec8869d7c1d7. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:41.925410 22404 consensus_queue.cc:1035] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:41.948611 22347 raft_consensus.cc:2932] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:41.952515 21327 raft_consensus.cc:2932] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:41.970912 21235 raft_consensus.cc:2932] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:41.998925 21243 consensus_queue.cc:237] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:42.038479 21037 catalog_manager.cc:5495] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
W20260812 19:12:42.143607 21204 consensus_peers.cc:487] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9548455d4eee4c74a47bf4b912e5740d. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:42.147505 21327 raft_consensus.cc:1261] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:42.150899 22441 consensus_queue.cc:1035] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:42.162601 21156 raft_consensus.cc:1261] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:42.165504 22441 consensus_queue.cc:1035] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:42.179199 22444 raft_consensus.cc:2932] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:42.285405 21327 raft_consensus.cc:2932] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:42.308754 21037 catalog_manager.cc:5495] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:42.352517 22465 ts_tablet_manager.cc:927] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:42.412967 22465 tablet_copy_client.cc:316] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.131:33285
W20260812 19:12:42.419158 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 47668.8us
I20260812 19:12:42.417426 21327 consensus_queue.cc:237] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:42.445621 21342 tablet_copy_service.cc:140] P 93578836968649f6bfcaecc709871412: Received BeginTabletCopySession request for tablet e6eaa29a6fe94976a6124729431bd8c8 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:50042)
I20260812 19:12:42.446363 21342 tablet_copy_service.cc:161] P 93578836968649f6bfcaecc709871412: Beginning new tablet copy session on tablet e6eaa29a6fe94976a6124729431bd8c8 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:50042: session id = 63db10063a7d4eb2a9cc9dc952654706-e6eaa29a6fe94976a6124729431bd8c8
W20260812 19:12:42.449036 21288 consensus_peers.cc:487] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eadd5203f7ca4c04b02bd28bea6b591c. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:42.457726 21342 tablet_copy_source_session.cc:205] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:42.459450 21244 raft_consensus.cc:1261] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:42.461395 22469 ts_tablet_manager.cc:927] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551)
I20260812 19:12:42.462035 22465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6eaa29a6fe94976a6124729431bd8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:42.464238 22355 consensus_queue.cc:1035] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:42.471326 22469 tablet_copy_client.cc:316] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.129:36551
I20260812 19:12:42.474341 21169 tablet_copy_service.cc:140] P f741102bfe1441d1b453e5a838649059: Received BeginTabletCopySession request for tablet dfa25b5175f14002a8b5ec8869d7c1d7 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:51004)
I20260812 19:12:42.475057 21169 tablet_copy_service.cc:161] P f741102bfe1441d1b453e5a838649059: Beginning new tablet copy session on tablet dfa25b5175f14002a8b5ec8869d7c1d7 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:51004: session id = 63db10063a7d4eb2a9cc9dc952654706-dfa25b5175f14002a8b5ec8869d7c1d7
I20260812 19:12:42.482060 22465 tablet_copy_client.cc:791] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:42.482990 22465 tablet_copy_client.cc:655] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:42.486863 21169 tablet_copy_source_session.cc:205] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:42.493019 22465 tablet_copy_client.cc:531] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:42.493777 22469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa25b5175f14002a8b5ec8869d7c1d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:42.500504 21152 raft_consensus.cc:2932] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:42.508518 22344 raft_consensus.cc:2932] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:42.510455 21235 raft_consensus.cc:2932] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:42.525708 22465 tablet_bootstrap.cc:492] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:42.539506 21153 raft_consensus.cc:1261] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:42.538794 21037 catalog_manager.cc:5495] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:42.557457 22469 tablet_copy_client.cc:791] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:42.558354 22469 tablet_copy_client.cc:655] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:42.563651 22469 tablet_copy_client.cc:531] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:42.580665 22344 consensus_queue.cc:1035] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:42.590045 22469 tablet_bootstrap.cc:492] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:42.640566 22473 ts_tablet_manager.cc:927] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:42.645548 21154 raft_consensus.cc:2932] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:42.655582 22473 tablet_copy_client.cc:316] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.130:33807
I20260812 19:12:42.657971 21254 tablet_copy_service.cc:140] P d1e1da0b9585460e8c365bf55679d4fa: Received BeginTabletCopySession request for tablet 9548455d4eee4c74a47bf4b912e5740d from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:57682)
I20260812 19:12:42.658571 21254 tablet_copy_service.cc:161] P d1e1da0b9585460e8c365bf55679d4fa: Beginning new tablet copy session on tablet 9548455d4eee4c74a47bf4b912e5740d from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:57682: session id = 63db10063a7d4eb2a9cc9dc952654706-9548455d4eee4c74a47bf4b912e5740d
I20260812 19:12:42.664358 22465 tablet_bootstrap.cc:492] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:42.665594 22465 tablet_bootstrap.cc:492] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:42.666438 22465 ts_tablet_manager.cc:1397] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.141s user 0.021s sys 0.007s
I20260812 19:12:42.669998 22465 raft_consensus.cc:348] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:42.671237 22465 raft_consensus.cc:729] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:42.672104 22465 consensus_queue.cc:260] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:42.681092 21254 tablet_copy_source_session.cc:205] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:42.683177 22465 ts_tablet_manager.cc:1428] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260812 19:12:42.685292 22473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9548455d4eee4c74a47bf4b912e5740d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:42.693148 22469 tablet_bootstrap.cc:492] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:42.694284 22469 tablet_bootstrap.cc:492] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:42.702044 22469 ts_tablet_manager.cc:1397] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.112s user 0.034s sys 0.008s
I20260812 19:12:42.706331 22469 raft_consensus.cc:348] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:42.708595 22469 raft_consensus.cc:729] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:42.709355 22469 consensus_queue.cc:260] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:42.716332 21342 tablet_copy_service.cc:342] P 93578836968649f6bfcaecc709871412: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-e6eaa29a6fe94976a6124729431bd8c8 received from {username='slave'} at 127.0.0.1:50042
I20260812 19:12:42.717062 21342 tablet_copy_service.cc:433] P 93578836968649f6bfcaecc709871412: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-e6eaa29a6fe94976a6124729431bd8c8 on tablet e6eaa29a6fe94976a6124729431bd8c8 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:42.731873 22473 tablet_copy_client.cc:791] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:42.732834 22473 tablet_copy_client.cc:655] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:42.735646 22053 raft_consensus.cc:1203] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:42.742249 22473 tablet_copy_client.cc:531] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:42.762176 22473 tablet_bootstrap.cc:492] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:42.786633 22469 ts_tablet_manager.cc:1428] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.084s user 0.020s sys 0.046s
I20260812 19:12:42.791591 21169 tablet_copy_service.cc:342] P f741102bfe1441d1b453e5a838649059: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-dfa25b5175f14002a8b5ec8869d7c1d7 received from {username='slave'} at 127.0.0.1:51004
I20260812 19:12:42.792069 21169 tablet_copy_service.cc:433] P f741102bfe1441d1b453e5a838649059: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-dfa25b5175f14002a8b5ec8869d7c1d7 on tablet dfa25b5175f14002a8b5ec8869d7c1d7 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:42.804441 22473 tablet_bootstrap.cc:492] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:42.805521 22473 tablet_bootstrap.cc:492] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:42.806414 22473 ts_tablet_manager.cc:1397] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.044s user 0.020s sys 0.013s
I20260812 19:12:42.809645 22473 raft_consensus.cc:348] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:42.810828 22473 raft_consensus.cc:729] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:42.811646 22473 consensus_queue.cc:260] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:42.820881 22473 ts_tablet_manager.cc:1428] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20260812 19:12:42.826073 21254 tablet_copy_service.cc:342] P d1e1da0b9585460e8c365bf55679d4fa: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-9548455d4eee4c74a47bf4b912e5740d received from {username='slave'} at 127.0.0.1:57682
I20260812 19:12:42.826555 21254 tablet_copy_service.cc:433] P d1e1da0b9585460e8c365bf55679d4fa: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-9548455d4eee4c74a47bf4b912e5740d on tablet 9548455d4eee4c74a47bf4b912e5740d with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:42.868738 22473 ts_tablet_manager.cc:927] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:42.888775 22473 tablet_copy_client.cc:316] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.131:33285
I20260812 19:12:42.893846 21342 tablet_copy_service.cc:140] P 93578836968649f6bfcaecc709871412: Received BeginTabletCopySession request for tablet eadd5203f7ca4c04b02bd28bea6b591c from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:50042)
I20260812 19:12:42.894753 21342 tablet_copy_service.cc:161] P 93578836968649f6bfcaecc709871412: Beginning new tablet copy session on tablet eadd5203f7ca4c04b02bd28bea6b591c from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:50042: session id = 63db10063a7d4eb2a9cc9dc952654706-eadd5203f7ca4c04b02bd28bea6b591c
I20260812 19:12:42.909425 21342 tablet_copy_source_session.cc:205] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:42.913528 22473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eadd5203f7ca4c04b02bd28bea6b591c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:42.941021 22473 tablet_copy_client.cc:791] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:42.941844 22473 tablet_copy_client.cc:655] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:42.949308 22473 tablet_copy_client.cc:531] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:42.961273 22473 tablet_bootstrap.cc:492] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:42.992981 22473 tablet_bootstrap.cc:492] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:42.994050 22473 tablet_bootstrap.cc:492] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:42.994942 22473 ts_tablet_manager.cc:1397] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.034s user 0.028s sys 0.006s
I20260812 19:12:42.998147 22473 raft_consensus.cc:348] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:42.999153 22473 raft_consensus.cc:729] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:42.999835 22473 consensus_queue.cc:260] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:43.006142 22473 ts_tablet_manager.cc:1428] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20260812 19:12:43.008204 21342 tablet_copy_service.cc:342] P 93578836968649f6bfcaecc709871412: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-eadd5203f7ca4c04b02bd28bea6b591c received from {username='slave'} at 127.0.0.1:50042
I20260812 19:12:43.008759 21342 tablet_copy_service.cc:433] P 93578836968649f6bfcaecc709871412: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-eadd5203f7ca4c04b02bd28bea6b591c on tablet eadd5203f7ca4c04b02bd28bea6b591c with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:43.027058 22056 raft_consensus.cc:1203] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:43.219533 22355 raft_consensus.cc:1050] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:43.221778 22355 consensus_queue.cc:237] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:43.242404 22056 raft_consensus.cc:1203] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:43.242895 22053 raft_consensus.cc:1261] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:43.244216 21235 raft_consensus.cc:1261] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:43.254189 22326 consensus_queue.cc:1035] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:43.269358 22442 consensus_queue.cc:1035] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:43.277855 21153 raft_consensus.cc:1261] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:43.286260 22442 consensus_queue.cc:1035] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:43.299371 22355 raft_consensus.cc:2932] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.302464 21244 raft_consensus.cc:2932] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.339170 21152 raft_consensus.cc:2932] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.352283 22053 raft_consensus.cc:2932] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.354501 21036 catalog_manager.cc:5495] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.488588 22056 raft_consensus.cc:1203] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:43.524725 21327 consensus_queue.cc:237] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:43.531096 22056 raft_consensus.cc:1261] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:43.533584 22442 consensus_queue.cc:1035] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:43.542430 21243 raft_consensus.cc:1261] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:43.542261 22344 raft_consensus.cc:2932] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.553262 22326 consensus_queue.cc:1035] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:43.553735 22056 raft_consensus.cc:2932] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.574365 21019 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e6eaa29a6fe94976a6124729431bd8c8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:43.564334 21243 raft_consensus.cc:2932] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.578111 22455 raft_consensus.cc:1050] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:43.581274 22455 consensus_queue.cc:237] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:43.596316 22056 raft_consensus.cc:1261] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:43.596222 21034 catalog_manager.cc:5495] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.598848 21327 raft_consensus.cc:1261] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:43.601449 21243 raft_consensus.cc:1261] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:43.608745 22463 consensus_queue.cc:1035] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:43.611786 22456 consensus_queue.cc:1035] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:43.614215 22456 consensus_queue.cc:1035] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260812 19:12:43.643172 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet e6eaa29a6fe94976a6124729431bd8c8 with delete_type TABLET_DATA_TOMBSTONED (TS f741102bfe1441d1b453e5a838649059 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54786
I20260812 19:12:43.661510 22496 tablet_replica.cc:330] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059: stopping tablet replica
I20260812 19:12:43.663625 22496 raft_consensus.cc:2229] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:43.668020 22456 raft_consensus.cc:2932] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.683112 22496 raft_consensus.cc:2258] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:43.674048 21327 raft_consensus.cc:2932] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.687037 22496 ts_tablet_manager.cc:1905] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:43.690631 22056 raft_consensus.cc:2932] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.713737 21243 raft_consensus.cc:2932] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.717760 21034 catalog_manager.cc:5495] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.728063 22462 raft_consensus.cc:1050] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:43.731115 22462 consensus_queue.cc:237] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:43.736676 22496 ts_tablet_manager.cc:1918] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:43.737355 22496 log.cc:1196] T e6eaa29a6fe94976a6124729431bd8c8 P f741102bfe1441d1b453e5a838649059: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-0-root/wals/e6eaa29a6fe94976a6124729431bd8c8
I20260812 19:12:43.740566 21022 catalog_manager.cc:4841] TS f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): tablet e6eaa29a6fe94976a6124729431bd8c8 (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:43.768095 21320 raft_consensus.cc:1261] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:43.769789 21153 raft_consensus.cc:1261] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:43.771623 22316 consensus_queue.cc:1035] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:43.772699 22460 consensus_queue.cc:1035] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:43.779369 22056 raft_consensus.cc:1261] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:43.789750 22316 consensus_queue.cc:1035] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:43.820281 21152 consensus_queue.cc:237] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:43.832706 22462 raft_consensus.cc:2932] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.839859 21328 raft_consensus.cc:1261] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:43.841688 22456 consensus_queue.cc:1035] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:12:43.844559 22056 raft_consensus.cc:1261] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:43.849411 22455 consensus_queue.cc:1035] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:43.852118 21320 raft_consensus.cc:2932] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.861001 22493 raft_consensus.cc:2932] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.865927 21327 raft_consensus.cc:2932] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.848482 21153 raft_consensus.cc:2932] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.890679 21036 catalog_manager.cc:5495] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.892622 22056 raft_consensus.cc:2932] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.896160 22055 raft_consensus.cc:2932] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.927187 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet dfa25b5175f14002a8b5ec8869d7c1d7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:43.927238 21034 catalog_manager.cc:5495] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.938735 21244 consensus_queue.cc:237] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:43.947767 22055 raft_consensus.cc:1261] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:43.948527 21152 raft_consensus.cc:1261] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:43.949606 22460 consensus_queue.cc:1035] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:43.952773 22460 consensus_queue.cc:1035] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:43.956877 21219 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa25b5175f14002a8b5ec8869d7c1d7 with delete_type TABLET_DATA_TOMBSTONED (TS d1e1da0b9585460e8c365bf55679d4fa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37890
I20260812 19:12:43.974416 22502 tablet_replica.cc:330] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa: stopping tablet replica
I20260812 19:12:43.973567 21153 raft_consensus.cc:2932] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.975698 22502 raft_consensus.cc:2229] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:43.976384 22502 raft_consensus.cc:2258] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:43.980428 22502 ts_tablet_manager.cc:1905] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:43.971653 22460 raft_consensus.cc:2932] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:44.015444 21037 catalog_manager.cc:5495] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 reported cstate change: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:43.971732 22055 raft_consensus.cc:2932] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:44.036788 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9548455d4eee4c74a47bf4b912e5740d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:44.030808 22502 ts_tablet_manager.cc:1918] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:44.038411 22502 log.cc:1196] T dfa25b5175f14002a8b5ec8869d7c1d7 P d1e1da0b9585460e8c365bf55679d4fa: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-1-root/wals/dfa25b5175f14002a8b5ec8869d7c1d7
I20260812 19:12:44.045899 21022 catalog_manager.cc:4841] TS d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): tablet dfa25b5175f14002a8b5ec8869d7c1d7 (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:44.086870 22344 raft_consensus.cc:1050] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:44.090960 21316 tablet_service.cc:1514] Processing DeleteTablet for tablet 9548455d4eee4c74a47bf4b912e5740d with delete_type TABLET_DATA_TOMBSTONED (TS 93578836968649f6bfcaecc709871412 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51536
I20260812 19:12:44.089373 22344 consensus_queue.cc:237] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:44.113057 22504 tablet_replica.cc:330] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412: stopping tablet replica
I20260812 19:12:44.114396 22504 raft_consensus.cc:2229] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:44.115326 22504 raft_consensus.cc:2258] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:44.120874 21244 raft_consensus.cc:1261] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:44.121909 22055 raft_consensus.cc:1261] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:44.123369 22344 consensus_queue.cc:1035] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:44.123955 21153 raft_consensus.cc:1261] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:44.125846 22355 consensus_queue.cc:1035] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:44.128556 22442 consensus_queue.cc:1035] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:44.133695 22504 ts_tablet_manager.cc:1905] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:44.195392 22355 raft_consensus.cc:2932] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:44.209241 22055 raft_consensus.cc:2932] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:44.228472 21154 raft_consensus.cc:2932] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:44.274053 21243 raft_consensus.cc:2932] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:44.276835 21036 catalog_manager.cc:5495] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:44.300367 22504 ts_tablet_manager.cc:1918] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:44.302317 22504 log.cc:1196] T 9548455d4eee4c74a47bf4b912e5740d P 93578836968649f6bfcaecc709871412: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-2-root/wals/9548455d4eee4c74a47bf4b912e5740d
I20260812 19:12:44.298509 21320 consensus_queue.cc:237] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:44.328924 21019 catalog_manager.cc:4841] TS 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): tablet 9548455d4eee4c74a47bf4b912e5740d (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:44.332839 21244 raft_consensus.cc:1261] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:44.346761 22511 consensus_queue.cc:1035] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:12:44.362474 22056 raft_consensus.cc:1261] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:44.367084 22511 consensus_queue.cc:1035] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:12:44.372100 22510 raft_consensus.cc:2932] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:44.384052 22056 raft_consensus.cc:2932] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:44.393596 21244 raft_consensus.cc:2932] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:44.405840 21019 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet eadd5203f7ca4c04b02bd28bea6b591c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:44.405370 21036 catalog_manager.cc:5495] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:44.443691 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet eadd5203f7ca4c04b02bd28bea6b591c with delete_type TABLET_DATA_TOMBSTONED (TS f741102bfe1441d1b453e5a838649059 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54786
I20260812 19:12:44.452732 22512 tablet_replica.cc:330] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059: stopping tablet replica
I20260812 19:12:44.454213 22512 raft_consensus.cc:2229] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:44.455222 22512 raft_consensus.cc:2258] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:44.458952 22512 ts_tablet_manager.cc:1905] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:44.481400 22512 ts_tablet_manager.cc:1918] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:44.481884 22512 log.cc:1196] T eadd5203f7ca4c04b02bd28bea6b591c P f741102bfe1441d1b453e5a838649059: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-0-root/wals/eadd5203f7ca4c04b02bd28bea6b591c
I20260812 19:12:44.485529 21022 catalog_manager.cc:4841] TS f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): tablet eadd5203f7ca4c04b02bd28bea6b591c (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:45.576536 21154 consensus_queue.cc:237] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:45.591113 21328 raft_consensus.cc:1261] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 19:12:45.592489 21115 consensus_peers.cc:487] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ce6040ab9fc41539e7fc09b3f9d05dd. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:45.593410 22513 consensus_queue.cc:1035] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:45.595265 21243 raft_consensus.cc:1261] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:45.596971 22513 consensus_queue.cc:1035] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:45.627948 22455 raft_consensus.cc:2932] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:45.629180 21328 raft_consensus.cc:2932] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:45.631187 21244 raft_consensus.cc:2932] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:45.652475 21037 catalog_manager.cc:5495] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:45.698822 21233 consensus_queue.cc:237] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:45.718075 21153 raft_consensus.cc:1261] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 19:12:45.720407 21204 consensus_peers.cc:487] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45e03c303e40432e8f5da639f7edff82. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:45.722378 22316 consensus_queue.cc:1035] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:45.722631 21328 raft_consensus.cc:1261] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:45.737658 22514 consensus_queue.cc:1035] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260812 19:12:45.749221 22514 raft_consensus.cc:2932] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:45.756316 21153 raft_consensus.cc:2932] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:45.767247 21320 raft_consensus.cc:2932] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:45.785091 21034 catalog_manager.cc:5495] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:45.819268 21321 consensus_queue.cc:237] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:45.861955 21233 raft_consensus.cc:1261] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:45.863793 22355 consensus_queue.cc:1035] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:45.864796 21154 raft_consensus.cc:1261] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 19:12:45.865573 21288 consensus_peers.cc:487] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db5ddd3c989548e5a199029ccf42040e. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:45.866403 22355 consensus_queue.cc:1035] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:45.881711 22519 raft_consensus.cc:2932] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:45.896489 21244 raft_consensus.cc:2932] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:45.907878 21154 raft_consensus.cc:2932] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:45.921933 21321 consensus_queue.cc:237] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:45.929899 21036 catalog_manager.cc:5495] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260812 19:12:45.988266 21288 consensus_peers.cc:487] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 932fac0d61e94da3a0f9c11ae2f597c3. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:45.990814 21233 raft_consensus.cc:1261] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:45.992442 21153 raft_consensus.cc:1261] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:45.993932 22355 consensus_queue.cc:1035] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:46.015484 22511 consensus_queue.cc:1035] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:46.036805 22519 raft_consensus.cc:2932] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:46.039626 21152 raft_consensus.cc:2932] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:46.067276 21244 raft_consensus.cc:2932] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:46.071336 21036 catalog_manager.cc:5495] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:46.130179 22542 ts_tablet_manager.cc:927] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551)
I20260812 19:12:46.140993 22542 tablet_copy_client.cc:316] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.129:36551
I20260812 19:12:46.142998 21169 tablet_copy_service.cc:140] P f741102bfe1441d1b453e5a838649059: Received BeginTabletCopySession request for tablet 9ce6040ab9fc41539e7fc09b3f9d05dd from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:51004)
I20260812 19:12:46.143599 21169 tablet_copy_service.cc:161] P f741102bfe1441d1b453e5a838649059: Beginning new tablet copy session on tablet 9ce6040ab9fc41539e7fc09b3f9d05dd from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:51004: session id = 63db10063a7d4eb2a9cc9dc952654706-9ce6040ab9fc41539e7fc09b3f9d05dd
I20260812 19:12:46.153311 21169 tablet_copy_source_session.cc:205] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:46.159101 22542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ce6040ab9fc41539e7fc09b3f9d05dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:46.185679 22542 tablet_copy_client.cc:791] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:46.186519 22542 tablet_copy_client.cc:655] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:46.191924 22542 tablet_copy_client.cc:531] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:46.201046 22544 ts_tablet_manager.cc:927] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:46.206717 22542 tablet_bootstrap.cc:492] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:46.218271 22544 tablet_copy_client.cc:316] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.130:33807
I20260812 19:12:46.220759 21254 tablet_copy_service.cc:140] P d1e1da0b9585460e8c365bf55679d4fa: Received BeginTabletCopySession request for tablet 45e03c303e40432e8f5da639f7edff82 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:57682)
I20260812 19:12:46.221645 21254 tablet_copy_service.cc:161] P d1e1da0b9585460e8c365bf55679d4fa: Beginning new tablet copy session on tablet 45e03c303e40432e8f5da639f7edff82 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:57682: session id = 63db10063a7d4eb2a9cc9dc952654706-45e03c303e40432e8f5da639f7edff82
I20260812 19:12:46.235913 21254 tablet_copy_source_session.cc:205] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:46.244350 22544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e03c303e40432e8f5da639f7edff82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:46.246392 22542 tablet_bootstrap.cc:492] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:46.247450 22542 tablet_bootstrap.cc:492] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:46.248543 22542 ts_tablet_manager.cc:1397] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.042s user 0.030s sys 0.007s
I20260812 19:12:46.251807 22542 raft_consensus.cc:348] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:46.252821 22542 raft_consensus.cc:729] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:46.253613 22542 consensus_queue.cc:260] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:46.266067 22542 ts_tablet_manager.cc:1428] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.017s user 0.012s sys 0.001s
I20260812 19:12:46.268715 21169 tablet_copy_service.cc:342] P f741102bfe1441d1b453e5a838649059: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-9ce6040ab9fc41539e7fc09b3f9d05dd received from {username='slave'} at 127.0.0.1:51004
I20260812 19:12:46.269210 21169 tablet_copy_service.cc:433] P f741102bfe1441d1b453e5a838649059: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-9ce6040ab9fc41539e7fc09b3f9d05dd on tablet 9ce6040ab9fc41539e7fc09b3f9d05dd with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:46.302454 22544 tablet_copy_client.cc:791] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:46.303206 22544 tablet_copy_client.cc:655] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:46.308662 22544 tablet_copy_client.cc:531] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:46.326702 22544 tablet_bootstrap.cc:492] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:46.359998 22544 tablet_bootstrap.cc:492] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:46.363353 22544 tablet_bootstrap.cc:492] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:46.364318 22544 ts_tablet_manager.cc:1397] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.038s user 0.019s sys 0.012s
I20260812 19:12:46.370100 22544 raft_consensus.cc:348] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:46.371346 22544 raft_consensus.cc:729] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:46.372259 22544 consensus_queue.cc:260] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:46.379523 22544 ts_tablet_manager.cc:1428] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260812 19:12:46.386906 21254 tablet_copy_service.cc:342] P d1e1da0b9585460e8c365bf55679d4fa: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-45e03c303e40432e8f5da639f7edff82 received from {username='slave'} at 127.0.0.1:57682
I20260812 19:12:46.387393 21254 tablet_copy_service.cc:433] P d1e1da0b9585460e8c365bf55679d4fa: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-45e03c303e40432e8f5da639f7edff82 on tablet 45e03c303e40432e8f5da639f7edff82 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:46.470866 22544 ts_tablet_manager.cc:927] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:46.476980 22544 tablet_copy_client.cc:316] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.131:33285
I20260812 19:12:46.480754 21342 tablet_copy_service.cc:140] P 93578836968649f6bfcaecc709871412: Received BeginTabletCopySession request for tablet db5ddd3c989548e5a199029ccf42040e from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:50042)
I20260812 19:12:46.481518 21342 tablet_copy_service.cc:161] P 93578836968649f6bfcaecc709871412: Beginning new tablet copy session on tablet db5ddd3c989548e5a199029ccf42040e from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:50042: session id = 63db10063a7d4eb2a9cc9dc952654706-db5ddd3c989548e5a199029ccf42040e
I20260812 19:12:46.493012 21342 tablet_copy_source_session.cc:205] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:46.497972 22544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5ddd3c989548e5a199029ccf42040e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:46.528354 22544 tablet_copy_client.cc:791] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:46.529141 22544 tablet_copy_client.cc:655] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:46.533671 22544 tablet_copy_client.cc:531] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:46.542614 22544 tablet_bootstrap.cc:492] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:46.582597 22055 raft_consensus.cc:1203] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:46.603390 22542 ts_tablet_manager.cc:927] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:46.645190 22542 tablet_copy_client.cc:316] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.131:33285
I20260812 19:12:46.648939 21342 tablet_copy_service.cc:140] P 93578836968649f6bfcaecc709871412: Received BeginTabletCopySession request for tablet 932fac0d61e94da3a0f9c11ae2f597c3 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:50042)
I20260812 19:12:46.649539 21342 tablet_copy_service.cc:161] P 93578836968649f6bfcaecc709871412: Beginning new tablet copy session on tablet 932fac0d61e94da3a0f9c11ae2f597c3 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:50042: session id = 63db10063a7d4eb2a9cc9dc952654706-932fac0d61e94da3a0f9c11ae2f597c3
I20260812 19:12:46.658660 22544 tablet_bootstrap.cc:492] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:46.661554 22544 tablet_bootstrap.cc:492] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:46.659340 21342 tablet_copy_source_session.cc:205] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:46.662698 22544 ts_tablet_manager.cc:1397] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.120s user 0.031s sys 0.049s
I20260812 19:12:46.667520 22544 raft_consensus.cc:348] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:46.668241 22542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 932fac0d61e94da3a0f9c11ae2f597c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:46.668267 22544 raft_consensus.cc:729] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:46.670109 22544 consensus_queue.cc:260] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:46.673424 22544 ts_tablet_manager.cc:1428] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260812 19:12:46.687469 21342 tablet_copy_service.cc:342] P 93578836968649f6bfcaecc709871412: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-db5ddd3c989548e5a199029ccf42040e received from {username='slave'} at 127.0.0.1:50042
I20260812 19:12:46.687958 21342 tablet_copy_service.cc:433] P 93578836968649f6bfcaecc709871412: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-db5ddd3c989548e5a199029ccf42040e on tablet db5ddd3c989548e5a199029ccf42040e with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:46.701258 22055 raft_consensus.cc:1203] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:46.775524 22542 tablet_copy_client.cc:791] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:46.776347 22542 tablet_copy_client.cc:655] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:46.782027 22542 tablet_copy_client.cc:531] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:46.799948 22542 tablet_bootstrap.cc:492] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:46.829272 22542 tablet_bootstrap.cc:492] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:46.831594 22542 tablet_bootstrap.cc:492] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:46.832520 22542 ts_tablet_manager.cc:1397] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.000s
I20260812 19:12:46.835973 22542 raft_consensus.cc:348] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:46.837677 22542 raft_consensus.cc:729] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:46.838747 22542 consensus_queue.cc:260] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:46.842410 22542 ts_tablet_manager.cc:1428] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260812 19:12:46.846953 21342 tablet_copy_service.cc:342] P 93578836968649f6bfcaecc709871412: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-932fac0d61e94da3a0f9c11ae2f597c3 received from {username='slave'} at 127.0.0.1:50042
I20260812 19:12:46.847487 21342 tablet_copy_service.cc:433] P 93578836968649f6bfcaecc709871412: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-932fac0d61e94da3a0f9c11ae2f597c3 on tablet 932fac0d61e94da3a0f9c11ae2f597c3 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:46.932614 22056 raft_consensus.cc:1203] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:47.109510 22056 raft_consensus.cc:1203] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:47.153995 22526 raft_consensus.cc:1050] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:47.156843 22526 consensus_queue.cc:237] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:47.177527 21320 raft_consensus.cc:1261] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:47.178799 22056 raft_consensus.cc:1261] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:47.181139 21244 raft_consensus.cc:1261] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:47.181532 22455 consensus_queue.cc:1035] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:47.184139 22455 consensus_queue.cc:1035] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:47.185731 22556 consensus_queue.cc:1035] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260812 19:12:47.303011 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 84535.5us
I20260812 19:12:47.299074 22493 raft_consensus.cc:2932] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.308323 22056 raft_consensus.cc:2932] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.317175 22517 raft_consensus.cc:1050] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:47.324600 22517 consensus_queue.cc:237] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:47.326596 21244 raft_consensus.cc:2932] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.344892 21320 raft_consensus.cc:2932] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.356261 21037 catalog_manager.cc:5495] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:47.367122 22056 raft_consensus.cc:1261] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:47.369165 22316 consensus_queue.cc:1035] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:47.379585 21320 raft_consensus.cc:1261] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:47.379654 21151 raft_consensus.cc:1261] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:47.382863 22566 consensus_queue.cc:1035] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:47.385391 22316 consensus_queue.cc:1035] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:12:47.406224 22563 raft_consensus.cc:1050] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:47.409165 22565 raft_consensus.cc:2932] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.410872 21151 raft_consensus.cc:2932] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.415252 22563 consensus_queue.cc:237] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:47.440858 21320 raft_consensus.cc:2932] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.444604 22056 raft_consensus.cc:2932] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.452768 22055 raft_consensus.cc:1261] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:47.455063 21244 raft_consensus.cc:1261] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:47.458037 22561 consensus_queue.cc:1035] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:47.459553 22519 consensus_queue.cc:1035] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:47.461548 21151 raft_consensus.cc:1261] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:47.463243 22511 consensus_queue.cc:1035] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:47.535938 22519 raft_consensus.cc:2932] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.541611 21151 raft_consensus.cc:2932] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.546005 22055 raft_consensus.cc:2932] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.560297 21152 consensus_queue.cc:237] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:47.569401 21244 raft_consensus.cc:2932] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.581995 21034 catalog_manager.cc:5495] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:47.590746 22056 raft_consensus.cc:1261] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:47.594148 22562 consensus_queue.cc:1035] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:47.602743 21320 raft_consensus.cc:1261] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:47.606009 22562 consensus_queue.cc:1035] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:47.608505 22493 raft_consensus.cc:2932] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.628433 21235 consensus_queue.cc:237] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:47.634137 22055 raft_consensus.cc:2932] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.628584 21036 catalog_manager.cc:5495] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.639429 21328 raft_consensus.cc:2932] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.641018 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9ce6040ab9fc41539e7fc09b3f9d05dd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:47.644086 22056 raft_consensus.cc:1261] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:47.648773 22517 consensus_queue.cc:1035] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:47.652171 21152 raft_consensus.cc:1261] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:47.656344 22566 consensus_queue.cc:1035] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:12:47.660948 22316 raft_consensus.cc:2932] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.669368 22355 raft_consensus.cc:1050] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:47.675403 22355 consensus_queue.cc:237] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:47.678339 22056 raft_consensus.cc:2932] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.686540 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 45e03c303e40432e8f5da639f7edff82 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:47.691155 21035 catalog_manager.cc:5495] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 reported cstate change: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:47.694108 21153 raft_consensus.cc:2932] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.695024 21037 catalog_manager.cc:5495] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:47.713562 21219 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ce6040ab9fc41539e7fc09b3f9d05dd with delete_type TABLET_DATA_TOMBSTONED (TS d1e1da0b9585460e8c365bf55679d4fa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37890
I20260812 19:12:47.731518 22056 raft_consensus.cc:1261] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:47.737439 22519 consensus_queue.cc:1035] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:47.740387 21244 raft_consensus.cc:1261] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:47.739418 21153 raft_consensus.cc:1261] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:47.744801 22561 consensus_queue.cc:1035] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:47.746881 22511 consensus_queue.cc:1035] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:47.753093 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:47.765086 22582 tablet_replica.cc:330] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa: stopping tablet replica
I20260812 19:12:47.767611 21316 tablet_service.cc:1514] Processing DeleteTablet for tablet 45e03c303e40432e8f5da639f7edff82 with delete_type TABLET_DATA_TOMBSTONED (TS 93578836968649f6bfcaecc709871412 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51536
I20260812 19:12:47.776729 22582 raft_consensus.cc:2229] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:47.777725 22582 raft_consensus.cc:2258] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:47.779752 22586 tablet_replica.cc:330] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412: stopping tablet replica
I20260812 19:12:47.790007 22586 raft_consensus.cc:2229] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:47.791190 22586 raft_consensus.cc:2258] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:47.772672 21321 consensus_queue.cc:237] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:47.788307 22582 ts_tablet_manager.cc:1905] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:47.806962 22056 raft_consensus.cc:1261] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:47.812095 22519 consensus_queue.cc:1035] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:47.807438 22586 ts_tablet_manager.cc:1905] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:47.836694 22511 raft_consensus.cc:2932] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.837306 22563 raft_consensus.cc:2932] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.841483 22056 raft_consensus.cc:2932] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.848282 21152 raft_consensus.cc:1261] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:47.868425 22051 raft_consensus.cc:2932] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.873643 21036 catalog_manager.cc:5495] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.889698 21219 tablet_service.cc:1514] Processing DeleteTablet for tablet db5ddd3c989548e5a199029ccf42040e with delete_type TABLET_DATA_TOMBSTONED (TS d1e1da0b9585460e8c365bf55679d4fa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37890
I20260812 19:12:47.905035 22561 consensus_queue.cc:1035] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:47.911422 22582 ts_tablet_manager.cc:1918] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:47.913314 22582 log.cc:1196] T 9ce6040ab9fc41539e7fc09b3f9d05dd P d1e1da0b9585460e8c365bf55679d4fa: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-1-root/wals/9ce6040ab9fc41539e7fc09b3f9d05dd
I20260812 19:12:47.916306 21022 catalog_manager.cc:4841] TS d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): tablet 9ce6040ab9fc41539e7fc09b3f9d05dd (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:47.917018 22582 tablet_replica.cc:330] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa: stopping tablet replica
I20260812 19:12:47.917995 22582 raft_consensus.cc:2229] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:47.926604 22582 raft_consensus.cc:2258] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:47.938185 22586 ts_tablet_manager.cc:1918] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:47.945120 22582 ts_tablet_manager.cc:1905] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:47.945488 22586 log.cc:1196] T 45e03c303e40432e8f5da639f7edff82 P 93578836968649f6bfcaecc709871412: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-2-root/wals/45e03c303e40432e8f5da639f7edff82
I20260812 19:12:47.949581 21019 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet db5ddd3c989548e5a199029ccf42040e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:47.954203 21244 raft_consensus.cc:2932] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.958983 21019 catalog_manager.cc:4841] TS 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): tablet 45e03c303e40432e8f5da639f7edff82 (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:47.966755 21153 raft_consensus.cc:2932] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.968415 22582 ts_tablet_manager.cc:1918] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:47.970772 21151 raft_consensus.cc:2932] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.984674 22582 log.cc:1196] T db5ddd3c989548e5a199029ccf42040e P d1e1da0b9585460e8c365bf55679d4fa: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-1-root/wals/db5ddd3c989548e5a199029ccf42040e
I20260812 19:12:47.984730 21036 catalog_manager.cc:5495] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:47.988056 21022 catalog_manager.cc:4841] TS d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): tablet db5ddd3c989548e5a199029ccf42040e (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:48.014205 21244 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:12:48.015588 21244 raft_consensus.cc:595] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:12:48.068516 21328 consensus_queue.cc:237] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:48.075533 21321 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf"
dest_uuid: "93578836968649f6bfcaecc709871412"
mode: GRACEFUL
new_leader_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:59192
I20260812 19:12:48.076246 22051 raft_consensus.cc:1261] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:48.076275 21321 raft_consensus.cc:595] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Received request to transfer leadership to TS f741102bfe1441d1b453e5a838649059
I20260812 19:12:48.078020 22591 consensus_queue.cc:1035] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:48.080552 21328 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7"
dest_uuid: "93578836968649f6bfcaecc709871412"
mode: GRACEFUL
new_leader_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:59192
I20260812 19:12:48.081176 21328 raft_consensus.cc:595] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Received request to transfer leadership to TS f741102bfe1441d1b453e5a838649059
I20260812 19:12:48.083812 21233 raft_consensus.cc:1261] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:48.089848 22591 consensus_queue.cc:1035] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:48.092938 22592 raft_consensus.cc:2932] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:48.100394 22051 raft_consensus.cc:2932] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:48.110107 21233 raft_consensus.cc:2932] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:48.122648 21019 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 932fac0d61e94da3a0f9c11ae2f597c3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:48.135295 22591 raft_consensus.cc:979] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412: : Instructing follower f741102bfe1441d1b453e5a838649059 to start an election
I20260812 19:12:48.135890 22591 raft_consensus.cc:1067] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Signalling peer f741102bfe1441d1b453e5a838649059 to start an election
I20260812 19:12:48.140482 21151 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:54796
I20260812 19:12:48.141192 21151 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:48.141571 21151 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:48.144671 21036 catalog_manager.cc:5495] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:48.152459 21151 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:48.155982 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:48.156765 21328 raft_consensus.cc:3032] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:48.157097 21328 raft_consensus.cc:729] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:48.157871 21328 consensus_queue.cc:260] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:48.159035 21328 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:48.167708 21151 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:48.167513 21328 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:12:48.170126 21112 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:48.171162 22493 raft_consensus.cc:2781] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:48.172178 22493 raft_consensus.cc:686] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:48.173218 22493 consensus_queue.cc:237] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:48.185438 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet 932fac0d61e94da3a0f9c11ae2f597c3 with delete_type TABLET_DATA_TOMBSTONED (TS f741102bfe1441d1b453e5a838649059 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54786
I20260812 19:12:48.196553 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:48.200485 21233 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:48.211556 21233 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:12:48.219138 21153 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:12:48.221611 22517 raft_consensus.cc:979] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa: : Instructing follower 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:12:48.222496 22517 raft_consensus.cc:1067] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Signalling peer 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:12:48.224416 21321 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c"
dest_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:51552
I20260812 19:12:48.224980 21321 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:48.225307 21321 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:12:48.225651 21153 raft_consensus.cc:595] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:12:48.227074 22596 tablet_replica.cc:330] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059: stopping tablet replica
I20260812 19:12:48.228858 22596 raft_consensus.cc:2229] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:48.230034 22596 raft_consensus.cc:2258] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:48.236316 21320 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98"
dest_uuid: "93578836968649f6bfcaecc709871412"
mode: GRACEFUL
new_leader_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:59192
I20260812 19:12:48.237058 21320 raft_consensus.cc:595] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Received request to transfer leadership to TS 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:48.236223 21321 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:48.241585 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:48.241816 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:48.243767 21153 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:12:48.243829 21243 raft_consensus.cc:3032] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:12:48.244838 21243 raft_consensus.cc:729] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:48.245872 21243 consensus_queue.cc:260] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:48.251376 22596 ts_tablet_manager.cc:1905] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:48.251983 22526 raft_consensus.cc:979] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059: : Instructing follower 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:12:48.252575 22562 raft_consensus.cc:1067] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Signalling peer 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:12:48.254734 21327 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416"
dest_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:51568
I20260812 19:12:48.254935 21153 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:12:48.255383 21327 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:48.255784 21327 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:48.256444 21287 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:48.257350 22591 raft_consensus.cc:2781] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:12:48.257885 22591 raft_consensus.cc:686] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:48.258876 22591 consensus_queue.cc:237] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:48.261771 21243 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:12:48.239522 21321 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:48.242748 21320 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c"
dest_uuid: "93578836968649f6bfcaecc709871412"
mode: GRACEFUL
new_leader_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:59192
I20260812 19:12:48.272327 21320 raft_consensus.cc:595] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Received request to transfer leadership to TS f741102bfe1441d1b453e5a838649059
I20260812 19:12:48.276966 21320 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e"
dest_uuid: "93578836968649f6bfcaecc709871412"
mode: GRACEFUL
new_leader_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:59192
I20260812 19:12:48.277657 21320 raft_consensus.cc:595] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Received request to transfer leadership to TS 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:48.279036 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260812 19:12:48.280906 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:12:48.281769 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:48.286996 22596 ts_tablet_manager.cc:1918] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:48.287549 22596 log.cc:1196] T 932fac0d61e94da3a0f9c11ae2f597c3 P f741102bfe1441d1b453e5a838649059: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-0-root/wals/932fac0d61e94da3a0f9c11ae2f597c3
I20260812 19:12:48.290135 21022 catalog_manager.cc:4841] TS f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): tablet 932fac0d61e94da3a0f9c11ae2f597c3 (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:48.241945 21036 catalog_manager.cc:5495] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 1 to 2, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 2 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:48.298203 21035 catalog_manager.cc:5495] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 4 to 5, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 5 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:48.301610 21327 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:48.303629 21327 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:48.307750 22563 raft_consensus.cc:979] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412: : Instructing follower 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:48.308014 22519 raft_consensus.cc:979] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412: : Instructing follower 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:48.308212 21243 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:12:48.308331 22563 raft_consensus.cc:1067] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Signalling peer 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:48.308557 22519 raft_consensus.cc:1067] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Signalling peer 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:48.322691 22055 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e"
dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:40614
I20260812 19:12:48.323324 22055 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:48.323690 22055 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:48.330900 22051 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98"
dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:40614
I20260812 19:12:48.331634 22051 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:48.332069 22051 raft_consensus.cc:3037] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:48.341938 22051 raft_consensus.cc:504] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:48.347860 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:48.348506 21153 raft_consensus.cc:3037] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:48.349632 22055 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:12:48.352064 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 49339.4us
I20260812 19:12:48.353571 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:48.354194 21152 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:48.356257 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:48.356896 21151 raft_consensus.cc:3032] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:48.357163 21151 raft_consensus.cc:729] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:48.357919 21151 consensus_queue.cc:260] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:48.373607 22055 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:48.376339 21151 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:48.378456 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:48.379153 21243 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:48.386025 21152 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:48.393757 21153 raft_consensus.cc:2454] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:48.398442 22011 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:48.399566 22548 raft_consensus.cc:2781] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:48.399806 22051 leader_election.cc:288] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:48.401773 22011 leader_election.cc:302] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:48.402896 21151 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:48.387596 21243 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:48.389618 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:48.404842 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:48.405786 21321 raft_consensus.cc:3032] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:48.406198 21321 raft_consensus.cc:729] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:48.407428 21321 consensus_queue.cc:260] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:48.409325 21321 raft_consensus.cc:3037] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:48.419025 21321 raft_consensus.cc:2454] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:48.404824 21327 raft_consensus.cc:3032] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:48.421729 21327 raft_consensus.cc:729] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:48.422755 21327 consensus_queue.cc:260] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:48.424314 21327 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:48.428925 21287 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:48.430831 22592 raft_consensus.cc:2781] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:48.431497 22592 raft_consensus.cc:686] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:48.432817 22592 consensus_queue.cc:237] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:48.437428 22548 raft_consensus.cc:686] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:12:48.440167 21327 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:48.441139 22600 raft_consensus.cc:2781] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:48.442269 22600 raft_consensus.cc:686] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:12:48.442533 22548 consensus_queue.cc:237] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:48.444111 22600 consensus_queue.cc:237] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:48.499603 21035 catalog_manager.cc:5495] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 1 to 2, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 2 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:48.508026 21036 catalog_manager.cc:5495] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 1 to 2, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 2 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:48.505151 21035 catalog_manager.cc:5495] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 1 to 2, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 2 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:48.529944 22591 raft_consensus.cc:979] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412: : Instructing follower f741102bfe1441d1b453e5a838649059 to start an election
I20260812 19:12:48.530483 22519 raft_consensus.cc:1067] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Signalling peer f741102bfe1441d1b453e5a838649059 to start an election
I20260812 19:12:48.532737 21152 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:54796
I20260812 19:12:48.533421 21152 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:48.533809 21152 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:48.552171 21152 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:48.558180 21152 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:48.558858 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:48.559859 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:48.560526 21328 raft_consensus.cc:3032] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:48.560809 21328 raft_consensus.cc:729] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:48.561686 21328 consensus_queue.cc:260] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:48.563164 21328 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:48.570847 21243 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:48.573371 21328 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:12:48.574908 21112 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:48.575857 22602 raft_consensus.cc:2781] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:48.576350 22602 raft_consensus.cc:686] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:48.577756 21243 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:12:48.577325 22602 consensus_queue.cc:237] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:48.591866 21036 catalog_manager.cc:5495] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 1 to 2, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 2 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:48.647918 22592 raft_consensus.cc:979] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412: : Instructing follower f741102bfe1441d1b453e5a838649059 to start an election
I20260812 19:12:48.648535 22519 raft_consensus.cc:1067] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Signalling peer f741102bfe1441d1b453e5a838649059 to start an election
I20260812 19:12:48.650120 21156 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:54796
I20260812 19:12:48.650729 21156 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:48.651081 21156 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:48.656359 21156 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:48.659421 21156 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:48.660578 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:48.661305 21243 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:48.660578 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:48.663266 21328 raft_consensus.cc:3032] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:12:48.663601 21328 raft_consensus.cc:729] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:48.664489 21328 consensus_queue.cc:260] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:48.665903 21328 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:48.667558 21243 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:12:48.668931 21114 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:48.669701 22602 raft_consensus.cc:2781] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:12:48.670169 22602 raft_consensus.cc:686] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:48.671618 22602 consensus_queue.cc:237] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:48.675082 21328 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:12:48.680291 21328 raft_consensus.cc:1261] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:48.682134 22556 consensus_queue.cc:1035] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:48.686833 21035 catalog_manager.cc:5495] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 2 to 3, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:48.701054 21233 raft_consensus.cc:1261] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260812 19:12:48.705665 22561 consensus_queue.cc:1035] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:48.766170 21243 raft_consensus.cc:1261] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:48.767987 22556 consensus_queue.cc:1035] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:48.781941 21156 raft_consensus.cc:1261] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260812 19:12:48.783922 22592 consensus_queue.cc:1035] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260812 19:12:48.906049 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 98304.4us
I20260812 19:12:48.907584 21321 raft_consensus.cc:1261] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:48.918753 21152 raft_consensus.cc:1261] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:48.923607 22600 consensus_queue.cc:1035] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:48.938808 22600 consensus_queue.cc:1035] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:48.948128 21152 raft_consensus.cc:1261] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:48.975986 21243 raft_consensus.cc:1261] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:48.981719 21328 raft_consensus.cc:1261] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:48.983350 22556 consensus_queue.cc:1035] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:48.998399 22519 consensus_queue.cc:1035] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:49.014701 22561 consensus_queue.cc:1035] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:49.021934 21320 raft_consensus.cc:1261] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:49.032652 22600 consensus_queue.cc:1035] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:49.039625 21156 raft_consensus.cc:1261] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:49.042867 22600 consensus_queue.cc:1035] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:49.056476 21243 raft_consensus.cc:1261] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:49.058854 22526 consensus_queue.cc:1035] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:49.150421 21233 raft_consensus.cc:1261] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260812 19:12:49.160763 22526 consensus_queue.cc:1035] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:49.276552 21321 raft_consensus.cc:1261] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260812 19:12:49.307214 22556 consensus_queue.cc:1035] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:49.403184 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:49.523103 21233 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3f11f84389864fddb8350b0f94829938"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:12:49.524137 21233 raft_consensus.cc:595] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Received request to transfer leadership to TS 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:49.527791 21233 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:12:49.528499 21233 raft_consensus.cc:595] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Received request to transfer leadership to TS 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:49.531968 21327 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3"
dest_uuid: "93578836968649f6bfcaecc709871412"
mode: GRACEFUL
new_leader_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:59192
I20260812 19:12:49.532681 21327 raft_consensus.cc:595] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Received request to transfer leadership to TS 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:49.540041 21152 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:12:49.540766 21152 raft_consensus.cc:595] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Received request to transfer leadership to TS 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:49.545704 21327 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9"
dest_uuid: "93578836968649f6bfcaecc709871412"
mode: GRACEFUL
new_leader_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:59192
I20260812 19:12:49.546412 21327 raft_consensus.cc:595] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Received request to transfer leadership to TS 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:49.549430 21321 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af"
dest_uuid: "93578836968649f6bfcaecc709871412"
mode: GRACEFUL
new_leader_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:59192
I20260812 19:12:49.550122 21321 raft_consensus.cc:595] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Received request to transfer leadership to TS f741102bfe1441d1b453e5a838649059
I20260812 19:12:49.553112 22591 raft_consensus.cc:979] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412: : Instructing follower 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:49.553972 22519 raft_consensus.cc:1067] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Signalling peer 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:49.554140 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260812 19:12:49.556689 22051 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9"
dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:40614
I20260812 19:12:49.557524 22051 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:49.558121 22051 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:49.559496 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:12:49.560559 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:49.577647 22051 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:49.582038 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:49.582221 22051 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:49.582424 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:49.582923 21151 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:49.583140 21327 raft_consensus.cc:3032] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:49.583489 21327 raft_consensus.cc:729] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:49.584606 21321 consensus_queue.cc:237] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:49.584496 21327 consensus_queue.cc:260] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:49.588586 21327 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:49.594039 21151 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:49.595741 22011 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:49.596910 22548 raft_consensus.cc:2781] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:49.597472 22548 raft_consensus.cc:686] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:12:49.598479 22548 consensus_queue.cc:237] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:49.602984 21327 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:49.614270 21233 raft_consensus.cc:1261] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:49.616320 22592 consensus_queue.cc:1035] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:49.622889 21151 raft_consensus.cc:1261] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 19:12:49.623952 21288 consensus_peers.cc:487] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad64c74a7de6449c8152bc0e093a59db. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:49.626096 22592 consensus_queue.cc:1035] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:49.637163 22561 raft_consensus.cc:979] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412: : Instructing follower 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:49.637724 22561 raft_consensus.cc:1067] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Signalling peer 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:49.646763 22056 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3"
dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:40614
I20260812 19:12:49.647503 22056 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:49.647890 22056 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:49.641774 21035 catalog_manager.cc:5495] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 1 to 2, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 2 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:49.672906 22519 raft_consensus.cc:2932] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:49.681835 21152 raft_consensus.cc:2932] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:49.695755 21151 consensus_queue.cc:237] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:49.678418 21233 raft_consensus.cc:2932] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:49.702365 21036 catalog_manager.cc:5495] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:49.707409 22056 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:49.723444 22056 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:49.724128 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:49.724803 21235 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:49.727175 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:49.727893 21327 raft_consensus.cc:3032] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:49.728204 21327 raft_consensus.cc:729] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:49.729027 21327 consensus_queue.cc:260] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:49.730530 21327 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:49.748737 21327 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:49.760504 21235 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:49.760749 22008 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:12:49.761827 22548 raft_consensus.cc:2781] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:49.768254 22548 raft_consensus.cc:686] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:12:49.770823 21327 raft_consensus.cc:1261] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:49.772513 22601 consensus_queue.cc:1035] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260812 19:12:49.773772 21115 consensus_peers.cc:487] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c13f33db56241a7938cfe52798e77a2. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:49.775238 21235 raft_consensus.cc:1261] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:49.769500 22548 consensus_queue.cc:237] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:49.785136 22556 consensus_queue.cc:1035] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:49.815037 22601 raft_consensus.cc:2932] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:49.869473 21035 catalog_manager.cc:5495] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 1 to 2, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 2 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:49.878897 21235 consensus_queue.cc:237] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:49.877525 21320 raft_consensus.cc:2932] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:49.888334 21037 catalog_manager.cc:5495] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:49.892753 21233 raft_consensus.cc:2932] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:49.902168 22601 raft_consensus.cc:979] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059: : Instructing follower 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:49.902359 21151 raft_consensus.cc:1261] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:49.902837 22601 raft_consensus.cc:1067] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Signalling peer 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:49.904202 22316 consensus_queue.cc:1035] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:49.905135 22051 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd"
dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:40624
I20260812 19:12:49.905845 22051 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:49.906345 22051 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
W20260812 19:12:49.913640 21204 consensus_peers.cc:487] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f5090bb13ea4aa792b27b04a7a79688. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:49.915031 22051 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:49.919421 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:49.920207 21152 raft_consensus.cc:3032] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:49.920502 21152 raft_consensus.cc:729] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:49.921372 21152 consensus_queue.cc:260] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:49.922593 21152 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:49.966291 21320 raft_consensus.cc:1261] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:49.977114 22612 raft_consensus.cc:979] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa: : Instructing follower 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:49.993068 22612 raft_consensus.cc:1067] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Signalling peer 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:49.994419 22316 consensus_queue.cc:1035] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260812 19:12:50.001096 22056 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459"
dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:40598
I20260812 19:12:50.001705 22056 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:50.002064 22056 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:49.990116 21152 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:50.003250 22051 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:50.003698 22011 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:50.004840 22548 raft_consensus.cc:2781] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:50.005451 22548 raft_consensus.cc:686] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:12:50.005678 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:50.006381 21320 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:50.006707 22548 consensus_queue.cc:237] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:50.015414 22612 raft_consensus.cc:2932] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:50.017503 21320 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:50.019949 21152 raft_consensus.cc:2932] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:50.029201 21151 raft_consensus.cc:1261] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:50.032119 22600 consensus_queue.cc:1035] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:50.044395 21034 catalog_manager.cc:5495] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:50.053659 22316 raft_consensus.cc:979] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa: : Instructing follower 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:50.054256 22316 raft_consensus.cc:1067] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Signalling peer 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:50.048095 22056 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:50.058538 22056 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:50.060220 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:50.060945 21321 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:50.064532 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:50.065231 21233 raft_consensus.cc:3032] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:50.065531 21233 raft_consensus.cc:729] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:50.067617 21035 catalog_manager.cc:5495] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 1 to 2, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 2 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:50.069171 21151 consensus_queue.cc:237] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:50.074328 21321 raft_consensus.cc:2454] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:50.066238 21233 consensus_queue.cc:260] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:50.076215 22008 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:12:50.076615 21233 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:50.081895 22519 raft_consensus.cc:979] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412: : Instructing follower f741102bfe1441d1b453e5a838649059 to start an election
I20260812 19:12:50.082484 22519 raft_consensus.cc:1067] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Signalling peer f741102bfe1441d1b453e5a838649059 to start an election
I20260812 19:12:50.086804 22056 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3f11f84389864fddb8350b0f94829938"
dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:40598
I20260812 19:12:50.089560 22056 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:50.090014 22056 raft_consensus.cc:3037] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:50.093683 21327 raft_consensus.cc:1261] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:50.104131 22602 consensus_queue.cc:1035] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:50.104760 21152 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:54796
I20260812 19:12:50.105423 21152 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:50.105881 21152 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:50.094668 22548 raft_consensus.cc:2781] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:50.110576 21320 raft_consensus.cc:2932] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:50.133767 21243 raft_consensus.cc:1261] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 19:12:50.134239 21115 consensus_peers.cc:487] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65db14a4838d40218947e8bd3420e892. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:50.133515 21233 raft_consensus.cc:2454] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:50.147837 22548 raft_consensus.cc:686] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
W20260812 19:12:50.123625 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 47050.5us
I20260812 19:12:50.138118 22526 consensus_queue.cc:1035] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260812 19:12:50.153914 22056 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:50.156103 22056 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:50.183094 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:50.188710 21243 raft_consensus.cc:3032] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:50.189038 21243 raft_consensus.cc:729] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:50.190294 22548 consensus_queue.cc:237] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:50.190594 21243 consensus_queue.cc:260] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:50.192204 21243 raft_consensus.cc:3037] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:50.196616 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:50.197312 21320 raft_consensus.cc:3037] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:50.197674 21321 raft_consensus.cc:1261] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:50.204972 22602 raft_consensus.cc:2932] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:50.207237 22662 consensus_queue.cc:1035] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:50.208586 21327 raft_consensus.cc:2932] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:50.218938 21152 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:50.220714 21243 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:50.232106 21320 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:50.235687 22008 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:12:50.236816 22662 raft_consensus.cc:2781] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:50.237429 22662 raft_consensus.cc:686] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:12:50.238448 22662 consensus_queue.cc:237] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:50.249305 22672 ts_tablet_manager.cc:927] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551)
I20260812 19:12:50.252008 21152 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:50.261005 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:50.262007 21321 raft_consensus.cc:3032] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:50.263932 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:50.266984 22668 ts_tablet_manager.cc:927] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:50.278972 21233 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:50.279117 22672 tablet_copy_client.cc:316] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.129:36551
I20260812 19:12:50.264681 21243 raft_consensus.cc:2932] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:50.273072 21034 catalog_manager.cc:5495] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:50.288240 21321 raft_consensus.cc:729] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:50.288106 21233 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:12:50.289301 21321 consensus_queue.cc:260] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:50.294447 21114 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:50.295774 22601 raft_consensus.cc:2781] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:50.296383 22601 raft_consensus.cc:686] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:50.297592 22601 consensus_queue.cc:237] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:50.299309 21035 catalog_manager.cc:5495] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 1 to 2, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 2 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:50.310475 21169 tablet_copy_service.cc:140] P f741102bfe1441d1b453e5a838649059: Received BeginTabletCopySession request for tablet 8c13f33db56241a7938cfe52798e77a2 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:51004)
I20260812 19:12:50.311326 21169 tablet_copy_service.cc:161] P f741102bfe1441d1b453e5a838649059: Beginning new tablet copy session on tablet 8c13f33db56241a7938cfe52798e77a2 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:51004: session id = 63db10063a7d4eb2a9cc9dc952654706-8c13f33db56241a7938cfe52798e77a2
I20260812 19:12:50.320154 21321 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:50.323879 21169 tablet_copy_source_session.cc:205] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:50.346769 21327 raft_consensus.cc:1261] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:50.349074 22662 consensus_queue.cc:1035] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:50.389952 21320 raft_consensus.cc:1261] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:50.392380 22662 consensus_queue.cc:1035] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:50.425968 21321 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:12:50.442481 22679 ts_tablet_manager.cc:927] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:50.441536 21035 catalog_manager.cc:5495] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 1 to 2, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 2 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:50.445639 22672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c13f33db56241a7938cfe52798e77a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:50.463115 22668 tablet_copy_client.cc:316] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.131:33285
I20260812 19:12:50.465126 21342 tablet_copy_service.cc:140] P 93578836968649f6bfcaecc709871412: Received BeginTabletCopySession request for tablet ad64c74a7de6449c8152bc0e093a59db from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:50042)
I20260812 19:12:50.465757 21342 tablet_copy_service.cc:161] P 93578836968649f6bfcaecc709871412: Beginning new tablet copy session on tablet ad64c74a7de6449c8152bc0e093a59db from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:50042: session id = 63db10063a7d4eb2a9cc9dc952654706-ad64c74a7de6449c8152bc0e093a59db
I20260812 19:12:50.467329 21152 raft_consensus.cc:1261] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:50.467661 21233 raft_consensus.cc:1261] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:50.470635 22679 tablet_copy_client.cc:316] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.130:33807
I20260812 19:12:50.471431 21037 catalog_manager.cc:5495] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 1 to 2, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 2 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:50.477006 22665 consensus_queue.cc:1035] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:50.483803 21342 tablet_copy_source_session.cc:205] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:50.486858 22665 consensus_queue.cc:1035] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:50.488986 21254 tablet_copy_service.cc:140] P d1e1da0b9585460e8c365bf55679d4fa: Received BeginTabletCopySession request for tablet 2f5090bb13ea4aa792b27b04a7a79688 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:57682)
I20260812 19:12:50.489722 21254 tablet_copy_service.cc:161] P d1e1da0b9585460e8c365bf55679d4fa: Beginning new tablet copy session on tablet 2f5090bb13ea4aa792b27b04a7a79688 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:57682: session id = 63db10063a7d4eb2a9cc9dc952654706-2f5090bb13ea4aa792b27b04a7a79688
I20260812 19:12:50.495532 22672 tablet_copy_client.cc:791] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:50.496394 22672 tablet_copy_client.cc:655] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:50.497187 22668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad64c74a7de6449c8152bc0e093a59db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:50.551505 21254 tablet_copy_source_session.cc:205] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:12:50.578137 22668 tablet_copy_client.cc:791] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:50.579216 22668 tablet_copy_client.cc:655] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:50.582718 22679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f5090bb13ea4aa792b27b04a7a79688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:50.585202 22668 tablet_copy_client.cc:531] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:50.591306 22672 tablet_copy_client.cc:531] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:50.596624 22668 tablet_bootstrap.cc:492] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:50.610337 22672 tablet_bootstrap.cc:492] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:50.628846 22679 tablet_copy_client.cc:791] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:50.660501 22679 tablet_copy_client.cc:655] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:50.650640 21243 raft_consensus.cc:1261] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:50.666180 22665 consensus_queue.cc:1035] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:50.668689 22686 ts_tablet_manager.cc:927] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551)
I20260812 19:12:50.709479 22679 tablet_copy_client.cc:531] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:50.717453 21327 raft_consensus.cc:1261] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:50.719341 22601 consensus_queue.cc:1035] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:50.720366 22679 tablet_bootstrap.cc:492] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:50.724241 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:50.765975 21328 raft_consensus.cc:1261] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:50.768927 22665 consensus_queue.cc:1035] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:50.774271 21233 raft_consensus.cc:1261] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:50.783723 22548 consensus_queue.cc:1035] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:12:50.793004 21321 raft_consensus.cc:1261] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:50.794862 22665 consensus_queue.cc:1035] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:12:50.818266 22668 tablet_bootstrap.cc:492] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:50.819625 22668 tablet_bootstrap.cc:492] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:50.819499 21233 raft_consensus.cc:1261] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:50.820596 22668 ts_tablet_manager.cc:1397] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.224s user 0.073s sys 0.065s
I20260812 19:12:50.831568 22668 raft_consensus.cc:348] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:50.832607 22668 raft_consensus.cc:729] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:50.833292 22668 consensus_queue.cc:260] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:50.841115 22668 ts_tablet_manager.cc:1428] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20260812 19:12:50.842883 21342 tablet_copy_service.cc:342] P 93578836968649f6bfcaecc709871412: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-ad64c74a7de6449c8152bc0e093a59db received from {username='slave'} at 127.0.0.1:50042
I20260812 19:12:50.843346 21342 tablet_copy_service.cc:433] P 93578836968649f6bfcaecc709871412: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-ad64c74a7de6449c8152bc0e093a59db on tablet ad64c74a7de6449c8152bc0e093a59db with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:50.867276 22679 tablet_bootstrap.cc:492] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:50.867962 22556 consensus_queue.cc:1035] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:50.868492 22679 tablet_bootstrap.cc:492] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:50.869449 22679 ts_tablet_manager.cc:1397] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.149s user 0.030s sys 0.036s
I20260812 19:12:50.874449 22679 raft_consensus.cc:348] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:50.875834 22679 raft_consensus.cc:729] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:50.876662 22679 consensus_queue.cc:260] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:50.879993 22679 ts_tablet_manager.cc:1428] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260812 19:12:50.883531 21254 tablet_copy_service.cc:342] P d1e1da0b9585460e8c365bf55679d4fa: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-2f5090bb13ea4aa792b27b04a7a79688 received from {username='slave'} at 127.0.0.1:57682
I20260812 19:12:50.884027 21254 tablet_copy_service.cc:433] P d1e1da0b9585460e8c365bf55679d4fa: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-2f5090bb13ea4aa792b27b04a7a79688 on tablet 2f5090bb13ea4aa792b27b04a7a79688 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:50.886861 22686 tablet_copy_client.cc:316] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.129:36551
I20260812 19:12:50.890882 21169 tablet_copy_service.cc:140] P f741102bfe1441d1b453e5a838649059: Received BeginTabletCopySession request for tablet 65db14a4838d40218947e8bd3420e892 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:51004)
I20260812 19:12:50.891585 21169 tablet_copy_service.cc:161] P f741102bfe1441d1b453e5a838649059: Beginning new tablet copy session on tablet 65db14a4838d40218947e8bd3420e892 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:51004: session id = 63db10063a7d4eb2a9cc9dc952654706-65db14a4838d40218947e8bd3420e892
I20260812 19:12:50.929136 21169 tablet_copy_source_session.cc:205] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059: Tablet Copy: opened 0 blocks and 1 log segments
W20260812 19:12:51.081936 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 53788.4us
I20260812 19:12:51.101706 22686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65db14a4838d40218947e8bd3420e892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:12:51.183650 22672 tablet_bootstrap.cc:492] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:51.186293 22672 tablet_bootstrap.cc:492] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:51.187295 22672 ts_tablet_manager.cc:1397] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.577s user 0.035s sys 0.014s
I20260812 19:12:51.203907 22672 raft_consensus.cc:348] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:51.204950 22672 raft_consensus.cc:729] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:51.205747 22672 consensus_queue.cc:260] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:51.208209 22672 ts_tablet_manager.cc:1428] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.020s user 0.004s sys 0.004s
I20260812 19:12:51.215099 22056 raft_consensus.cc:1203] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:51.224342 22686 tablet_copy_client.cc:791] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:12:51.225169 22686 tablet_copy_client.cc:655] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:12:51.316694 22056 raft_consensus.cc:1203] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:51.336462 21237 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:12:51.337169 21237 raft_consensus.cc:595] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:12:51.336639 21169 tablet_copy_service.cc:342] P f741102bfe1441d1b453e5a838649059: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-8c13f33db56241a7938cfe52798e77a2 received from {username='slave'} at 127.0.0.1:51004
I20260812 19:12:51.341372 21169 tablet_copy_service.cc:433] P f741102bfe1441d1b453e5a838649059: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-8c13f33db56241a7938cfe52798e77a2 on tablet 8c13f33db56241a7938cfe52798e77a2 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:51.344803 22709 raft_consensus.cc:979] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa: : Instructing follower 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:12:51.345486 22708 raft_consensus.cc:1067] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Signalling peer 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:12:51.348496 21327 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122"
dest_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:51552
I20260812 19:12:51.349066 22686 tablet_copy_client.cc:531] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:12:51.349161 21327 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:51.349647 21327 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:51.362123 21327 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:51.364187 21327 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:51.365690 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:51.366487 21237 raft_consensus.cc:3032] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:51.367010 21237 raft_consensus.cc:729] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:51.367937 21237 consensus_queue.cc:260] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:51.369477 21237 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:51.378214 21237 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:51.380136 21287 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:51.381131 22561 raft_consensus.cc:2781] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:51.381682 22561 raft_consensus.cc:686] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:51.383064 22561 consensus_queue.cc:237] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:51.385689 22686 tablet_bootstrap.cc:492] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:12:51.403452 22051 raft_consensus.cc:1203] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260812 19:12:51.442104 21154 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "65db14a4838d40218947e8bd3420e892"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:12:51.442950 21154 raft_consensus.cc:595] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Received request to transfer leadership to TS d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:51.463169 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:51.463992 21150 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:51.472851 21236 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:12:51.473694 21236 raft_consensus.cc:595] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:12:51.477609 21150 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:51.479454 21236 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:12:51.480232 21236 raft_consensus.cc:595] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Received request to transfer leadership to TS 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:51.483620 21150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:12:51.484288 21150 raft_consensus.cc:595] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Received request to transfer leadership to TS d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:51.486452 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260812 19:12:51.489465 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:12:51.490283 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:51.939294 22700 raft_consensus.cc:979] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059: : Instructing follower d1e1da0b9585460e8c365bf55679d4fa to start an election
I20260812 19:12:51.940287 22700 raft_consensus.cc:1067] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Signalling peer d1e1da0b9585460e8c365bf55679d4fa to start an election
W20260812 19:12:51.998529 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 63907.7us
I20260812 19:12:52.000756 22716 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:12:52.001479 22716 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:12:52.002857 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 492620us
I20260812 19:12:52.014022 22700 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:12:52.023366 21243 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65db14a4838d40218947e8bd3420e892"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:37902
I20260812 19:12:52.026821 21243 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:52.027262 21243 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:52.031013 22712 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:12:52.032624 22712 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:52.035388 21151 raft_consensus.cc:1261] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:52.039587 22712 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:52.040413 22716 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:52.041622 22591 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:12:52.042260 22591 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:52.044736 22591 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:52.048604 22667 consensus_queue.cc:1035] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:52.060365 22591 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:12:52.061102 22591 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:52.063392 22591 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:52.066988 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:52.050868 22700 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:52.068066 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:52.071074 22659 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:12:52.071830 22659 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:52.056490 21243 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:52.077584 21235 raft_consensus.cc:1261] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:52.109826 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:52.112934 21243 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:52.121423 22519 consensus_queue.cc:1035] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:52.129546 22659 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:52.100826 22700 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:52.142482 22686 tablet_bootstrap.cc:492] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:12:52.143860 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:52.144596 21322 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:52.144769 22686 tablet_bootstrap.cc:492] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:12:52.145598 22686 ts_tablet_manager.cc:1397] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 0.760s user 0.051s sys 0.052s
I20260812 19:12:52.148779 22686 raft_consensus.cc:348] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:52.149755 22686 raft_consensus.cc:729] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:12:52.150465 22686 consensus_queue.cc:260] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:52.151901 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:52.152797 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:52.153429 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:52.153934 22686 ts_tablet_manager.cc:1428] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260812 19:12:52.155604 22720 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:12:52.156265 22720 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:52.158694 22720 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:52.159179 21322 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:12:52.167202 22617 raft_consensus.cc:979] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa: : Instructing follower 93578836968649f6bfcaecc709871412 to start an election
W20260812 19:12:52.177320 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 130816us
I20260812 19:12:52.178025 22617 raft_consensus.cc:1067] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Signalling peer 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:12:52.168305 22519 raft_consensus.cc:1050] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:52.162583 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:52.179968 21200 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:52.181066 22519 consensus_queue.cc:237] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:52.183530 22556 raft_consensus.cc:1050] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:52.186383 22556 consensus_queue.cc:237] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:52.181069 22708 raft_consensus.cc:2781] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:52.194152 22708 raft_consensus.cc:686] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:52.195420 22711 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:12:52.196221 22711 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:52.201864 22711 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:52.205166 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:52.205636 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:52.205813 21320 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7"
dest_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:51552
I20260812 19:12:52.206411 21320 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:52.206877 21320 raft_consensus.cc:3037] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:52.209794 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:52.211381 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:52.216261 21320 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:52.195621 22708 consensus_queue.cc:237] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:52.273057 21153 raft_consensus.cc:1261] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:52.272003 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:52.263549 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:52.278101 21150 raft_consensus.cc:3037] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:52.252661 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:52.254364 22620 raft_consensus.cc:979] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa: : Instructing follower 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:52.218889 21320 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:52.279552 22612 raft_consensus.cc:1067] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Signalling peer 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:52.268507 21238 raft_consensus.cc:1261] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:52.255291 22055 raft_consensus.cc:1261] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:52.223374 21324 raft_consensus.cc:1226] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
I20260812 19:12:52.229247 21169 tablet_copy_service.cc:342] P f741102bfe1441d1b453e5a838649059: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-65db14a4838d40218947e8bd3420e892 received from {username='slave'} at 127.0.0.1:51004
I20260812 19:12:52.253146 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:52.282790 22711 consensus_queue.cc:1035] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:52.285138 22711 consensus_queue.cc:1035] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:12:52.251269 22011 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:52.288868 22659 raft_consensus.cc:2726] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:12:52.232685 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:12:52.239797 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 106166us
I20260812 19:12:52.220729 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:52.293804 21243 raft_consensus.cc:3032] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:52.294194 21243 raft_consensus.cc:729] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:52.294744 21169 tablet_copy_service.cc:433] P f741102bfe1441d1b453e5a838649059: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-65db14a4838d40218947e8bd3420e892 on tablet 65db14a4838d40218947e8bd3420e892 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:52.296478 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:52.297195 21154 raft_consensus.cc:3032] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:52.298947 21035 catalog_manager.cc:5495] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 1 to 2, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 2 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:52.297386 21243 consensus_queue.cc:260] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:52.306797 21243 raft_consensus.cc:3037] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:52.297510 21154 raft_consensus.cc:729] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:52.314594 21150 raft_consensus.cc:2454] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:52.310248 21154 consensus_queue.cc:260] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:52.317639 21154 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:52.321616 21324 raft_consensus.cc:1261] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:52.324992 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:52.332553 21243 raft_consensus.cc:2454] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:52.333979 21287 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:52.334993 22620 raft_consensus.cc:1050] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:52.335006 22711 raft_consensus.cc:2781] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:52.330333 22011 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:52.336424 22711 raft_consensus.cc:686] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:52.337821 22711 consensus_queue.cc:237] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:52.341141 22659 raft_consensus.cc:2726] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:12:52.345779 22051 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b"
dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:40598
I20260812 19:12:52.346822 22051 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:52.347478 22051 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:52.352352 21287 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:52.353262 22519 raft_consensus.cc:2726] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:12:52.353570 22053 raft_consensus.cc:1261] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:52.353660 21287 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:52.354482 22519 raft_consensus.cc:2726] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 19:12:52.356786 22721 log.cc:924] Time spent T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260812 19:12:52.360558 21236 raft_consensus.cc:1261] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:52.368479 22700 consensus_queue.cc:1035] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:52.373080 22718 consensus_queue.cc:1035] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260812 19:12:52.377463 21287 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:52.379274 22519 raft_consensus.cc:2726] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:12:52.379791 21287 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:52.381462 21287 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:12:52.383764 22561 raft_consensus.cc:2726] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:12:52.384518 21154 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:12:52.384841 22592 raft_consensus.cc:2726] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:12:52.406040 21112 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:52.339407 22620 consensus_queue.cc:237] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:12:52.413618 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 95893.6us
I20260812 19:12:52.413921 22700 raft_consensus.cc:2726] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:12:52.394433 22602 consensus_queue.cc:1035] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
W20260812 19:12:52.449501 21266 debug-util.cc:398] Leaking SignalData structure 0x7b080017d7c0 after lost signal to thread 19742
W20260812 19:12:52.450615 21266 debug-util.cc:398] Leaking SignalData structure 0x7b08001b6420 after lost signal to thread 21077
W20260812 19:12:52.452431 21266 debug-util.cc:398] Leaking SignalData structure 0x7b080034efa0 after lost signal to thread 21185
W20260812 19:12:52.453967 21266 debug-util.cc:398] Leaking SignalData structure 0x7b0800308f00 after lost signal to thread 21270
I20260812 19:12:52.418290 22051 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:52.456075 22602 consensus_queue.cc:1035] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
W20260812 19:12:52.464239 21266 debug-util.cc:398] Leaking SignalData structure 0x7b08002c9d40 after lost signal to thread 21359
I20260812 19:12:52.473667 22053 raft_consensus.cc:1226] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:12:52.472810 22714 raft_consensus.cc:979] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059: : Instructing follower d1e1da0b9585460e8c365bf55679d4fa to start an election
I20260812 19:12:52.476872 22601 raft_consensus.cc:1067] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Signalling peer d1e1da0b9585460e8c365bf55679d4fa to start an election
I20260812 19:12:52.478922 21236 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:37902
I20260812 19:12:52.479583 21236 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:52.480055 21236 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:52.483346 22519 raft_consensus.cc:2932] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
W20260812 19:12:52.481875 21266 debug-util.cc:398] Leaking SignalData structure 0x7b0800316740 after lost signal to thread 22084
I20260812 19:12:52.560475 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:52.561192 22056 raft_consensus.cc:2932] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
W20260812 19:12:52.587344 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 78835.4us
I20260812 19:12:52.613377 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:52.614167 21321 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:52.593281 22051 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:52.625653 21321 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:52.604764 22697 consensus_queue.cc:1046] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260812 19:12:52.577464 21238 raft_consensus.cc:2932] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:52.657840 21320 raft_consensus.cc:2932] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:52.630793 21236 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:12:52.660784 22735 log.cc:924] Time spent T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260812 19:12:52.661183 21236 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:12:52.662283 21235 raft_consensus.cc:3032] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:52.662708 21235 raft_consensus.cc:729] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:52.663578 21235 consensus_queue.cc:260] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:12:52.675707 22723 log.cc:924] Time spent T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.116s user 0.003s sys 0.002s
I20260812 19:12:52.676271 22051 raft_consensus.cc:1261] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 19:12:52.671567 22736 raft_consensus.cc:2932] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:52.679325 22709 consensus_queue.cc:1035] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
W20260812 19:12:52.738686 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 90181.8us
I20260812 19:12:52.741021 21324 raft_consensus.cc:1261] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260812 19:12:52.737195 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 123849us
I20260812 19:12:52.747457 22705 consensus_queue.cc:1035] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260812 19:12:52.752192 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:52.752929 21154 raft_consensus.cc:3032] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:52.753249 21154 raft_consensus.cc:729] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:52.754015 21154 consensus_queue.cc:260] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:52.766765 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:12:52.767486 22052 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:52.768621 22056 raft_consensus.cc:2932] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:52.784948 22052 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:12:52.798907 21204 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:52.802194 21150 raft_consensus.cc:1261] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260812 19:12:52.805897 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 53890.2us
I20260812 19:12:52.796694 21155 raft_consensus.cc:2932] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:52.811743 22612 consensus_queue.cc:1035] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:12:52.755793 22008 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:12:52.824193 21154 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:52.834741 21154 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
W20260812 19:12:52.841210 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 81505.5us
I20260812 19:12:52.840780 22612 raft_consensus.cc:2781] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:52.866796 22662 raft_consensus.cc:2781] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election won for term 2
W20260812 19:12:52.855753 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 100055us
I20260812 19:12:52.890833 21235 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:52.874574 22662 raft_consensus.cc:686] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:12:52.879482 22612 raft_consensus.cc:686] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:52.931540 22662 consensus_queue.cc:237] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:52.927979 22612 consensus_queue.cc:237] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:52.979920 21235 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
W20260812 19:12:52.982863 22697 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260812 19:12:52.987185 21035 catalog_manager.cc:5495] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:52.989838 21035 catalog_manager.cc:5495] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 1 to 2, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 2 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:52.997674 22051 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Advancing to term 2
I20260812 19:12:53.009305 22051 raft_consensus.cc:1261] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260812 19:12:53.011839 22707 consensus_queue.cc:1035] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:53.018780 21035 catalog_manager.cc:5495] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
W20260812 19:12:53.027479 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 145214us
I20260812 19:12:53.050303 21153 raft_consensus.cc:1261] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:53.055655 21156 raft_consensus.cc:1261] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260812 19:12:53.057538 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 230104us
I20260812 19:12:53.076996 22519 consensus_queue.cc:1035] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:53.089206 21327 raft_consensus.cc:1226] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:12:53.095700 21235 raft_consensus.cc:2932] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:53.114306 21243 raft_consensus.cc:1226] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
I20260812 19:12:53.091624 22617 consensus_queue.cc:1035] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260812 19:12:53.130985 21150 raft_consensus.cc:2932] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:53.119520 22707 raft_consensus.cc:2932] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
W20260812 19:12:53.196190 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 105918us
W20260812 19:12:53.260145 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 44473.3us
I20260812 19:12:53.278048 21237 raft_consensus.cc:1261] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:53.316521 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260812 19:12:53.339423 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 178935us
W20260812 19:12:53.435958 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 151613us
W20260812 19:12:53.436403 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 170220us
W20260812 19:12:53.462028 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 198981us
W20260812 19:12:53.464432 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 119717us
W20260812 19:12:53.474779 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 161031us
I20260812 19:12:53.493633 22052 raft_consensus.cc:2932] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:53.494432 21327 raft_consensus.cc:2932] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:53.488085 22592 consensus_queue.cc:1035] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260812 19:12:53.517026 21324 raft_consensus.cc:1261] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260812 19:12:53.518216 21328 raft_consensus.cc:1261] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:53.566596 22786 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:53.573668 21158 raft_consensus.cc:1261] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:53.586078 22764 consensus_queue.cc:1035] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:53.584168 22707 consensus_queue.cc:1035] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260812 19:12:53.599280 22784 raft_consensus.cc:1050] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
W20260812 19:12:53.599820 22784 raft_consensus.cc:1054] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa: Unable to promote non-voter 63db10063a7d4eb2a9cc9dc952654706: Illegal state: Leader has not yet committed an operation in its own term
I20260812 19:12:53.607214 21238 raft_consensus.cc:1261] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:53.619527 21158 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:12:53.620262 21158 raft_consensus.cc:595] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Received request to transfer leadership to TS d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:53.639843 22056 raft_consensus.cc:1261] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:53.637631 21238 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:12:53.644433 22707 consensus_queue.cc:1035] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:53.648013 21238 raft_consensus.cc:595] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:12:53.641114 22662 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260812 19:12:53.651970 21238 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:12:53.652819 21238 raft_consensus.cc:595] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:12:53.654788 22707 raft_consensus.cc:979] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa: : Instructing follower 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:12:53.655479 22707 raft_consensus.cc:1067] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Signalling peer 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:12:53.663723 21328 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "784d71ae37454b3a9e8023f268428468"
dest_uuid: "93578836968649f6bfcaecc709871412"
mode: GRACEFUL
new_leader_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:59192
I20260812 19:12:53.664336 21320 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b"
dest_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:51552
I20260812 19:12:53.664536 21328 raft_consensus.cc:595] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Received request to transfer leadership to TS 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:53.664935 21320 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:53.665364 21320 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:53.668043 21240 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:12:53.668936 21240 raft_consensus.cc:595] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Received request to transfer leadership to TS 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:53.670521 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260812 19:12:53.672324 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:12:53.679917 21320 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:53.686282 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:53.690442 21156 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:53.691659 21320 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:53.716058 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:53.727309 21034 catalog_manager.cc:5495] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 1 to 2, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 2 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:53.776155 21033 catalog_manager.cc:5495] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 1 to 2, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 2 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:53.782214 22519 raft_consensus.cc:979] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412: : Instructing follower 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:53.782802 22519 raft_consensus.cc:1067] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Signalling peer 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:53.784909 22051 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "784d71ae37454b3a9e8023f268428468"
dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:40614
I20260812 19:12:53.785583 22051 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:53.785950 22051 raft_consensus.cc:3037] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:53.786643 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:53.787460 21240 raft_consensus.cc:3032] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:53.787802 21240 raft_consensus.cc:729] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:53.788756 21240 consensus_queue.cc:260] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:53.794422 22051 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:53.820487 21240 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:53.872846 22700 raft_consensus.cc:979] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059: : Instructing follower d1e1da0b9585460e8c365bf55679d4fa to start an election
W20260812 19:12:53.900938 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 191158us
I20260812 19:12:53.927680 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:53.928740 22051 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:53.933313 21154 raft_consensus.cc:3037] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
W20260812 19:12:53.929873 22730 log.cc:924] Time spent T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1: Append to log took a long time: real 0.099s user 0.000s sys 0.001s
I20260812 19:12:53.957700 21240 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:53.958844 21320 consensus_queue.cc:237] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:53.967860 21287 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:53.985127 21154 raft_consensus.cc:2454] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:53.987121 22011 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:12:53.967428 22797 log.cc:924] Time spent T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.137s user 0.001s sys 0.000s
I20260812 19:12:54.002067 22781 raft_consensus.cc:2781] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election won for term 2
W20260812 19:12:54.019146 22794 log.cc:924] Time spent T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20260812 19:12:54.003974 22755 raft_consensus.cc:1067] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Signalling peer d1e1da0b9585460e8c365bf55679d4fa to start an election
I20260812 19:12:54.013303 22662 raft_consensus.cc:2781] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:54.051673 21238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:37902
I20260812 19:12:54.052472 21238 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
W20260812 19:12:54.055002 22793 log.cc:924] Time spent T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.121s user 0.002s sys 0.000s
I20260812 19:12:54.055140 21238 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:54.068297 21156 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
W20260812 19:12:54.087596 22795 log.cc:924] Time spent T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260812 19:12:54.067699 22781 raft_consensus.cc:686] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
W20260812 19:12:54.104679 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 136891us
W20260812 19:12:54.211081 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 179026us
I20260812 19:12:54.190296 22662 raft_consensus.cc:686] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:12:54.190418 21155 raft_consensus.cc:1226] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 1. Current term is 2. Ops: []
W20260812 19:12:54.256667 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 125174us
W20260812 19:12:54.227144 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 85657.9us
I20260812 19:12:54.304484 21238 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:54.140271 22781 consensus_queue.cc:237] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:54.428156 22055 raft_consensus.cc:1261] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260812 19:12:54.433292 22798 log.cc:924] Time spent T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.324s user 0.000s sys 0.001s
I20260812 19:12:54.330555 22662 consensus_queue.cc:237] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:12:54.489748 22800 log.cc:924] Time spent T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.259s user 0.000s sys 0.001s
I20260812 19:12:54.517434 21238 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:12:54.595296 22807 log.cc:924] Time spent T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.304s user 0.002s sys 0.000s
W20260812 19:12:54.596942 22792 log.cc:924] Time spent T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.682s user 0.004s sys 0.014s
W20260812 19:12:54.717242 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 245452us
W20260812 19:12:54.738816 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 479751us
I20260812 19:12:54.739054 22617 raft_consensus.cc:979] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: : Instructing follower 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:12:54.765281 22703 raft_consensus.cc:1067] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Signalling peer 93578836968649f6bfcaecc709871412 to start an election
W20260812 19:12:54.771206 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 316914us
W20260812 19:12:54.773929 21286 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.001s sys 0.015s
W20260812 19:12:54.777061 21286 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.543s user 0.001s sys 0.015s
I20260812 19:12:54.789534 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:54.791034 21321 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:12:54.778273 22716 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.006s sys 0.009s
I20260812 19:12:54.796268 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:54.782204 22592 consensus_queue.cc:1035] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260812 19:12:54.786917 22822 consensus_queue.cc:1046] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260812 19:12:54.798141 21237 raft_consensus.cc:1226] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 2. Current term is 3. Ops: []
I20260812 19:12:54.799532 22055 raft_consensus.cc:1226] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 1. Current term is 2. Ops: []
I20260812 19:12:54.816645 22703 raft_consensus.cc:979] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa: : Instructing follower 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:54.817247 22703 raft_consensus.cc:1067] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Signalling peer 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:54.820518 21321 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:12:54.824687 21321 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692"
dest_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:51552
I20260812 19:12:54.825263 21321 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:54.825711 21321 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
W20260812 19:12:54.803740 22716 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.538s user 0.007s sys 0.010s
I20260812 19:12:54.830807 22055 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0"
dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:40598
I20260812 19:12:54.831503 22055 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:54.831918 22055 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:54.834758 21321 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:54.837445 21321 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:54.845510 21200 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:54.847270 22777 raft_consensus.cc:2781] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:12:54.847846 22777 raft_consensus.cc:686] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:12:54.850550 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 307596us
I20260812 19:12:54.854835 21323 raft_consensus.cc:3032] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:54.855446 21323 raft_consensus.cc:729] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:54.856412 21323 consensus_queue.cc:260] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:54.863796 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:54.864494 21153 raft_consensus.cc:3032] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:12:54.864769 21153 raft_consensus.cc:729] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:54.865535 21153 consensus_queue.cc:260] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:54.866730 21153 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:54.868548 21034 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48434 (request call id 233) took 2026 ms. Trace:
I20260812 19:12:54.848923 22777 consensus_queue.cc:237] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:54.857821 21323 raft_consensus.cc:3037] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:54.880455 21153 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:12:54.893644 21324 raft_consensus.cc:1226] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:12:54.869025 21034 rpcz_store.cc:276] 0812 19:12:52.841797 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:52.841943 (+ 146us) service_pool.cc:227] Handling call
0812 19:12:54.868436 (+2026493us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0812 19:12:53.784026 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0812 19:12:53.784673 (+ 647us) write_op.cc:432] Acquiring schema lock in shared mode
0812 19:12:53.784704 (+ 31us) write_op.cc:435] Acquired schema lock
0812 19:12:53.784719 (+ 15us) tablet.cc:660] Decoding operations
0812 19:12:53.785057 (+ 338us) write_op.cc:620] Acquiring the partition lock for write op
0812 19:12:53.785117 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0812 19:12:53.785140 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0812 19:12:53.785330 (+ 190us) tablet.cc:699] Row locks acquired
0812 19:12:53.785343 (+ 13us) write_op.cc:260] PREPARE: finished
0812 19:12:53.785494 (+ 151us) write_op.cc:270] Start()
0812 19:12:53.785627 (+ 133us) write_op.cc:276] Timestamp: P: 1786561973785445 usec, L: 0
0812 19:12:53.785649 (+ 22us) op_driver.cc:331] REPLICATION: starting
0812 19:12:53.786275 (+ 626us) log.cc:842] Serialized 599 byte log entry
0812 19:12:54.066071 (+279796us) op_driver.cc:444] REPLICATION: finished
0812 19:12:54.084484 (+ 18413us) write_op.cc:301] APPLY: starting
0812 19:12:54.091095 (+ 6611us) tablet.cc:1365] starting BulkCheckPresence
0812 19:12:54.099997 (+ 8902us) tablet.cc:1368] finished BulkCheckPresence
0812 19:12:54.100142 (+ 145us) tablet.cc:1370] starting ApplyRowOperation cycle
0812 19:12:54.132424 (+ 32282us) tablet.cc:1381] finished ApplyRowOperation cycle
0812 19:12:54.143180 (+ 10756us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0812 19:12:54.190549 (+ 47369us) write_op.cc:312] APPLY: finished
0812 19:12:54.233089 (+ 42540us) log.cc:842] Serialized 23 byte log entry
0812 19:12:54.257173 (+ 24084us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:12:54.261727 (+ 4554us) write_op.cc:454] Released schema lock
0812 19:12:54.265644 (+ 3917us) write_op.cc:341] FINISH: Updating metrics
0812 19:12:54.291271 (+ 25627us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:12:54.296020 (+ 4749us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":51875,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":3401,"apply.run_cpu_time_us":3641,"apply.run_wall_time_us":233576,"num_ops":1,"prepare.queue_time_us":304,"prepare.run_cpu_time_us":2547,"prepare.run_wall_time_us":2546,"replication_time_us":263214}]]}
I20260812 19:12:54.901443 21236 raft_consensus.cc:1261] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:54.910074 22055 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:54.924643 22830 consensus_queue.cc:1046] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:54.926451 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:54.926329 22808 raft_consensus.cc:3032] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:54.927358 22808 raft_consensus.cc:729] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:54.928336 22808 consensus_queue.cc:260] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:54.929404 22808 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:54.931027 22519 raft_consensus.cc:2932] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:54.933285 22051 raft_consensus.cc:2932] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
W20260812 19:12:54.944015 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 52011.6us
I20260812 19:12:54.950264 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:54.927271 21155 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:54.954463 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:54.955546 21243 raft_consensus.cc:3032] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:54.955984 21243 raft_consensus.cc:729] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:54.959004 21243 consensus_queue.cc:260] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:54.959580 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:54.967310 21154 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:54.960754 22055 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:54.974577 21154 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:54.979441 21155 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:54.974243 21238 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:54.989104 21323 raft_consensus.cc:2454] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
W20260812 19:12:54.989953 22822 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260812 19:12:54.992434 21243 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:54.992319 22011 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:54.993667 22806 raft_consensus.cc:2781] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:54.994241 22806 raft_consensus.cc:686] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:12:54.995226 22806 consensus_queue.cc:237] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:55.000108 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:55.001334 22784 raft_consensus.cc:1050] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:12:55.004217 22784 consensus_queue.cc:237] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:55.005841 22519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:51536 (request call id 485) took 1275 ms. Trace:
I20260812 19:12:55.007545 22519 rpcz_store.cc:276] 0812 19:12:53.730441 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:53.761968 (+ 31527us) service_pool.cc:227] Handling call
0812 19:12:53.957712 (+195744us) spinlock_profiling.cc:243] Waited 55.2 ms on lock 0x7b5800043ab0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9a21f63 00007f52d9a85aba 00007f52d9aa1a7a 00007f52d9aa2ada 00007f52d9aa1c8b 00007f52d9aad084 00007f52d9aac743 00007f52d9aa4c1a 00007f52de9e86d9 00007f52d80de184
0812 19:12:54.059047 (+101335us) spinlock_profiling.cc:243] Waited 5.4 ms on lock 0x7b2c00112cb8. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d8495e11 00007f52d84a4ada 00007f52d84a4763 00007f52d84a4701 00007f52d9ae1601 00007f52d9a333ee 00007f52d9aa1ab9 00007f52d9aa2ada 00007f52d9aa1c8b 00007f52d9aad084 00007f52d9aac743
0812 19:12:54.060876 (+ 1829us) log.cc:842] Serialized 501 byte log entry
0812 19:12:54.809428 (+748552us) spinlock_profiling.cc:243] Waited 3.29 ms on lock 0x7b54002f7c20. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9a37a4f 00007f52d9a20545 00007f52d9a25771 00007f52d9a256f9 00007f52d9a25689 00007f52d9a25651 00007f52d9a2481d 00007f52dfde2204 00007f52dfde2039 00007f52d6b8a6b6
0812 19:12:55.005714 (+196286us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5307,"raft.queue_time_us":252217,"raft.run_cpu_time_us":11802,"raft.run_wall_time_us":597712,"spinlock_wait_cycles":146845056,"thread_start_us":250719,"threads_started":2,"wal-append.queue_time_us":194963}
I20260812 19:12:55.024068 22053 raft_consensus.cc:1261] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260812 19:12:55.026031 22777 consensus_queue.cc:1035] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:55.028821 21287 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:55.036576 21155 raft_consensus.cc:1261] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260812 19:12:55.038540 22822 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:55.039139 21236 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "65db14a4838d40218947e8bd3420e892"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37890
I20260812 19:12:55.039218 22822 raft_consensus.cc:686] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:55.039844 21236 raft_consensus.cc:595] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Received request to transfer leadership
I20260812 19:12:55.040014 21318 raft_consensus.cc:1261] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260812 19:12:55.042577 22814 consensus_queue.cc:1035] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:55.043509 22808 consensus_queue.cc:1035] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:12:55.044301 22822 consensus_queue.cc:237] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:55.046372 22777 consensus_queue.cc:1035] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:55.057459 21019 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ad64c74a7de6449c8152bc0e093a59db with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:55.070415 22784 raft_consensus.cc:979] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa: : Instructing follower 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:55.071451 22784 raft_consensus.cc:1067] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Signalling peer 63db10063a7d4eb2a9cc9dc952654706 to start an election
I20260812 19:12:55.073213 22052 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65db14a4838d40218947e8bd3420e892"
dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:40598
I20260812 19:12:55.073923 22052 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:55.074343 22052 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:55.085264 22784 raft_consensus.cc:2932] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:55.088384 21324 raft_consensus.cc:2932] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:55.099888 21033 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58998 (request call id 131) took 1985 ms. Trace:
I20260812 19:12:55.106030 21155 raft_consensus.cc:2932] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:55.107364 22052 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:55.109402 21243 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:12:55.111114 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:55.112643 21036 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48428 (request call id 230) took 1825 ms. Trace:
I20260812 19:12:55.114892 22808 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:12:55.122574 22808 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:55.124907 22808 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:55.126796 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:55.129258 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:55.115674 21238 raft_consensus.cc:2932] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:55.132009 22052 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:55.137061 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:55.140498 22011 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:55.141836 22806 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:12:55.143455 21235 raft_consensus.cc:3032] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:12:55.143940 21235 raft_consensus.cc:729] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:55.144831 21235 consensus_queue.cc:260] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:55.145605 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:55.146190 21235 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:55.147351 21203 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:12:55.148237 22826 raft_consensus.cc:2726] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:12:55.100396 21033 rpcz_store.cc:276] 0812 19:12:53.114895 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:53.116005 (+ 1110us) service_pool.cc:227] Handling call
0812 19:12:55.099808 (+1983803us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0812 19:12:53.962666 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0812 19:12:54.077895 (+115229us) write_op.cc:432] Acquiring schema lock in shared mode
0812 19:12:54.080858 (+ 2963us) write_op.cc:435] Acquired schema lock
0812 19:12:54.080894 (+ 36us) tablet.cc:660] Decoding operations
0812 19:12:54.127602 (+ 46708us) write_op.cc:620] Acquiring the partition lock for write op
0812 19:12:54.132489 (+ 4887us) write_op.cc:641] Partition lock acquired for write op
0812 19:12:54.133800 (+ 1311us) tablet.cc:683] Acquiring locks for 1 operations
0812 19:12:54.148308 (+ 14508us) tablet.cc:699] Row locks acquired
0812 19:12:54.190949 (+ 42641us) write_op.cc:260] PREPARE: finished
0812 19:12:54.230414 (+ 39465us) write_op.cc:270] Start()
0812 19:12:54.593721 (+363307us) spinlock_profiling.cc:243] Waited 2.05 ms on lock 0x7b5400002728. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dffc7e91 00007f52dffc6afc 00007f52dd4ca304 00007f52dd4b57e3 00007f52dd4b52b8 00007f52dd4b9ba1 00007f52dd4b9b59 00007f52dd4b9ae9 00007f52dd4b9ab1 00007f52dd4b8dad 00007f52dfde2204
0812 19:12:54.594097 (+ 376us) write_op.cc:276] Timestamp: P: 1786561974230362 usec, L: 0
0812 19:12:54.594133 (+ 36us) op_driver.cc:331] REPLICATION: starting
0812 19:12:54.600244 (+ 6111us) log.cc:842] Serialized 536 byte log entry
0812 19:12:54.842302 (+242058us) op_driver.cc:444] REPLICATION: finished
0812 19:12:55.007261 (+164959us) write_op.cc:301] APPLY: starting
0812 19:12:55.008530 (+ 1269us) tablet.cc:1365] starting BulkCheckPresence
0812 19:12:55.008637 (+ 107us) tablet.cc:1368] finished BulkCheckPresence
0812 19:12:55.008657 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0812 19:12:55.008907 (+ 250us) tablet.cc:1381] finished ApplyRowOperation cycle
0812 19:12:55.009068 (+ 161us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0812 19:12:55.009114 (+ 46us) write_op.cc:312] APPLY: finished
0812 19:12:55.009318 (+ 204us) log.cc:842] Serialized 23 byte log entry
0812 19:12:55.009485 (+ 167us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:12:55.009594 (+ 109us) write_op.cc:454] Released schema lock
0812 19:12:55.009632 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0812 19:12:55.095777 (+ 86145us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:12:55.095813 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":164654,"apply.run_cpu_time_us":1912,"apply.run_wall_time_us":88841,"num_ops":1,"prepare.queue_time_us":4682,"prepare.run_cpu_time_us":17087,"prepare.run_wall_time_us":655385,"replication_time_us":248037,"spinlock_wait_cycles":4708224,"thread_start_us":153805,"threads_started":1}]]}
I20260812 19:12:55.130004 21036 rpcz_store.cc:276] 0812 19:12:53.287042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:53.287226 (+ 184us) service_pool.cc:227] Handling call
0812 19:12:55.112574 (+1825348us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":3}
I20260812 19:12:55.120813 21156 consensus_queue.cc:237] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:55.159147 21235 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:12:55.159379 21154 raft_consensus.cc:1261] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:55.163986 21323 raft_consensus.cc:1261] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:55.167822 22052 raft_consensus.cc:1226] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: []
I20260812 19:12:55.175813 22806 consensus_queue.cc:1035] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:12:55.177274 22806 consensus_queue.cc:1035] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:12:55.190821 21034 catalog_manager.cc:5495] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index 2 to 3, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:55.194773 21034 catalog_manager.cc:5495] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 1 to 2, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 2 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:55.207787 21034 catalog_manager.cc:5495] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index 3 to 4, VOTER f741102bfe1441d1b453e5a838649059 (127.19.70.129) evicted. New cstate: current_term: 1 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:55.219272 22052 raft_consensus.cc:1261] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:55.221817 22779 consensus_queue.cc:1035] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:55.226923 21156 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:12:55.229290 21156 raft_consensus.cc:595] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Received request to transfer leadership to TS d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:55.243439 21155 raft_consensus.cc:1261] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:55.247910 21236 raft_consensus.cc:1261] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:55.271673 21321 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021"
dest_uuid: "93578836968649f6bfcaecc709871412"
mode: GRACEFUL
new_leader_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:59192
I20260812 19:12:55.272581 21321 raft_consensus.cc:595] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Received request to transfer leadership to TS f741102bfe1441d1b453e5a838649059
I20260812 19:12:55.272680 22824 consensus_queue.cc:1035] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:55.273650 22519 consensus_queue.cc:1035] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:55.279340 21324 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "784d71ae37454b3a9e8023f268428468"
dest_uuid: "93578836968649f6bfcaecc709871412"
mode: GRACEFUL
new_leader_uuid: "63db10063a7d4eb2a9cc9dc952654706"
from {username='slave'} at 127.0.0.1:59192
I20260812 19:12:55.280361 21324 raft_consensus.cc:595] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Received request to transfer leadership to TS 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:12:55.280895 21324 raft_consensus.cc:603] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:12:55.282477 21318 raft_consensus.cc:1261] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:55.285658 22779 consensus_queue.cc:1035] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:55.289291 21238 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:12:55.290277 21238 raft_consensus.cc:595] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:12:55.298698 21238 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:12:55.302949 21238 raft_consensus.cc:595] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:12:55.304651 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260812 19:12:55.306576 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:12:55.307380 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:55.334740 22777 raft_consensus.cc:979] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa: : Instructing follower 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:12:55.336987 22777 raft_consensus.cc:1067] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Signalling peer 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:12:55.358916 21324 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22"
dest_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:51552
I20260812 19:12:55.393582 22802 raft_consensus.cc:979] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa: : Instructing follower 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:12:55.394446 22705 raft_consensus.cc:1067] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Signalling peer 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:12:55.396827 21324 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:55.397351 21324 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:55.404578 21321 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f"
dest_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:51552
I20260812 19:12:55.434046 21321 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:55.434551 21321 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:55.442364 21319 raft_consensus.cc:1261] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:12:55.463027 21321 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:55.470871 21321 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:12:55.480111 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:12:55.480878 22056 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:55.493484 22820 raft_consensus.cc:2932] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:55.457615 22777 consensus_queue.cc:1035] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260812 19:12:55.497779 21318 raft_consensus.cc:2932] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:55.514374 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:55.515261 21236 raft_consensus.cc:3032] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:55.515626 21236 raft_consensus.cc:729] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:55.516439 21236 consensus_queue.cc:260] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:55.518007 21236 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:55.525713 22056 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:55.532583 21288 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:12:55.533566 22817 raft_consensus.cc:2781] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:55.534246 22817 raft_consensus.cc:686] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:55.535933 22817 consensus_queue.cc:237] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:55.540202 22779 raft_consensus.cc:979] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059: : Instructing follower d1e1da0b9585460e8c365bf55679d4fa to start an election
I20260812 19:12:55.540760 22779 raft_consensus.cc:1067] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Signalling peer d1e1da0b9585460e8c365bf55679d4fa to start an election
I20260812 19:12:55.543449 21233 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:37902
I20260812 19:12:55.544080 21233 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:55.544463 21233 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:55.556143 21233 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:55.555053 21036 catalog_manager.cc:5495] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index 2 to 4, term changed from 2 to 3, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to <none>, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:55.562114 21036 catalog_manager.cc:5495] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 1 to 2, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 2 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:55.566872 21235 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:55.572448 22766 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:12:55.571435 21036 catalog_manager.cc:5495] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 2 to 3, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 3 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:55.573094 22766 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:55.573900 22592 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260812 19:12:55.575832 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 102616us
I20260812 19:12:55.577805 21233 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:55.584604 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:55.584933 22766 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:12:55.585414 21323 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:55.589502 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:55.566030 21324 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:55.592860 21324 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:55.593336 21236 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:55.596347 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet ad64c74a7de6449c8152bc0e093a59db with delete_type TABLET_DATA_TOMBSTONED (TS f741102bfe1441d1b453e5a838649059 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54786
I20260812 19:12:55.601040 21150 raft_consensus.cc:1261] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:55.606155 22056 raft_consensus.cc:2932] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:55.611174 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:55.613554 21155 raft_consensus.cc:1261] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:12:55.614500 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:55.615496 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:55.617458 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:12:55.637856 21152 raft_consensus.cc:3032] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Stepping down as leader of term 1
W20260812 19:12:55.633615 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 52443.6us
I20260812 19:12:55.636353 21153 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:55.638408 21152 raft_consensus.cc:729] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:55.646124 22519 raft_consensus.cc:979] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412: : Instructing follower f741102bfe1441d1b453e5a838649059 to start an election
I20260812 19:12:55.646001 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:55.646593 22519 raft_consensus.cc:1067] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Signalling peer f741102bfe1441d1b453e5a838649059 to start an election
I20260812 19:12:55.648329 21236 raft_consensus.cc:3032] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:55.648758 21236 raft_consensus.cc:729] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:55.649572 21236 consensus_queue.cc:260] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:55.610162 22806 consensus_queue.cc:1035] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260812 19:12:55.652331 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 8c13f33db56241a7938cfe52798e77a2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:55.653695 21155 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:55.655314 22784 consensus_queue.cc:1035] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:12:55.656293 21156 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:54796
I20260812 19:12:55.656888 21156 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:55.657238 21156 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:55.653769 21115 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:55.658782 22766 raft_consensus.cc:2726] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:12:55.650703 21153 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:55.665093 22817 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:55.650771 21152 consensus_queue.cc:260] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:55.666101 21233 consensus_queue.cc:237] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:55.673897 21287 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:55.679673 21235 raft_consensus.cc:1261] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:55.680264 21152 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:55.681950 22592 raft_consensus.cc:2781] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:55.682523 22592 raft_consensus.cc:686] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:55.683506 22592 consensus_queue.cc:237] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:55.686172 22716 consensus_queue.cc:1035] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:55.700220 21156 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:55.705092 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:55.706076 21235 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:55.706867 21152 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:12:55.707952 21203 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:55.706101 21156 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:55.710755 22864 tablet_replica.cc:330] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059: stopping tablet replica
I20260812 19:12:55.711944 22864 raft_consensus.cc:2229] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:55.712792 22864 raft_consensus.cc:2258] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:55.714913 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:55.715622 21321 raft_consensus.cc:3032] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:55.715973 21321 raft_consensus.cc:729] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:55.719889 22777 raft_consensus.cc:2781] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:55.721582 21235 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:12:55.723489 21114 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:55.720602 22777 raft_consensus.cc:686] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:55.726996 21321 consensus_queue.cc:260] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:55.735679 22777 consensus_queue.cc:237] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:12:55.737687 21093 auto_rebalancer.cc:663] Could not move replica: Not found: Couldn't find leader for tablet 65db14a4838d40218947e8bd3420e892
W20260812 19:12:55.738127 21093 auto_rebalancer.cc:264] scheduled replica move failed to complete: Not found: Couldn't find leader for tablet 65db14a4838d40218947e8bd3420e892
I20260812 19:12:55.737454 22056 raft_consensus.cc:1261] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:55.743964 21152 raft_consensus.cc:1261] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260812 19:12:55.721202 21236 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:55.747761 21321 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
W20260812 19:12:55.755985 22866 log.cc:924] Time spent T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260812 19:12:55.756510 22864 ts_tablet_manager.cc:1905] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:55.744905 22859 raft_consensus.cc:2781] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:55.758316 22859 raft_consensus.cc:686] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:55.759501 22859 consensus_queue.cc:237] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:55.744733 21323 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:12:55.747790 22830 consensus_queue.cc:1035] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20260812 19:12:55.771381 22784 consensus_queue.cc:1035] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:55.807807 21033 catalog_manager.cc:5495] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 1 to 2, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 2 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:12:55.843397 21236 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:55.867421 21321 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:12:55.810025 21033 catalog_manager.cc:5495] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 1 to 2, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 2 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260812 19:12:56.019022 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 86368.1us
I20260812 19:12:56.067605 22830 raft_consensus.cc:2932] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:56.073675 22056 raft_consensus.cc:1261] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260812 19:12:56.078904 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 75977.8us
W20260812 19:12:56.079761 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 45081.2us
I20260812 19:12:56.078810 22740 consensus_queue.cc:1035] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
W20260812 19:12:56.103667 22872 log.cc:924] Time spent T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20260812 19:12:56.118156 22864 ts_tablet_manager.cc:1918] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:56.118778 22864 log.cc:1196] T ad64c74a7de6449c8152bc0e093a59db P f741102bfe1441d1b453e5a838649059: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-0-root/wals/ad64c74a7de6449c8152bc0e093a59db
I20260812 19:12:56.142369 21155 raft_consensus.cc:2932] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
W20260812 19:12:56.175891 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 112527us
W20260812 19:12:56.238638 22875 log.cc:924] Time spent T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
W20260812 19:12:56.284776 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 117813us
I20260812 19:12:56.287146 21022 catalog_manager.cc:4841] TS f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): tablet ad64c74a7de6449c8152bc0e093a59db (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:56.286590 22051 raft_consensus.cc:2932] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:56.309311 21036 catalog_manager.cc:5495] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 3 to 2, leader changed from <none> to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 2 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:56.313045 21036 catalog_manager.cc:5495] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 1 to 2, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 2 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:56.315014 21036 catalog_manager.cc:5495] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 1 to 2, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 2 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:56.320514 21033 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58998 (request call id 132) took 1121 ms. Trace:
I20260812 19:12:56.320704 21233 raft_consensus.cc:1261] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:56.321033 21033 rpcz_store.cc:276] 0812 19:12:55.199135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:55.209087 (+ 9952us) service_pool.cc:227] Handling call
0812 19:12:56.320439 (+1111352us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0812 19:12:55.996540 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0812 19:12:56.068803 (+ 72263us) write_op.cc:432] Acquiring schema lock in shared mode
0812 19:12:56.068858 (+ 55us) write_op.cc:435] Acquired schema lock
0812 19:12:56.068873 (+ 15us) tablet.cc:660] Decoding operations
0812 19:12:56.069339 (+ 466us) write_op.cc:620] Acquiring the partition lock for write op
0812 19:12:56.069398 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0812 19:12:56.069423 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0812 19:12:56.069793 (+ 370us) tablet.cc:699] Row locks acquired
0812 19:12:56.069809 (+ 16us) write_op.cc:260] PREPARE: finished
0812 19:12:56.069962 (+ 153us) write_op.cc:270] Start()
0812 19:12:56.070092 (+ 130us) write_op.cc:276] Timestamp: P: 1786561976069918 usec, L: 0
0812 19:12:56.070124 (+ 32us) op_driver.cc:331] REPLICATION: starting
0812 19:12:56.098815 (+ 28691us) log.cc:842] Serialized 919 byte log entry
0812 19:12:56.271570 (+172755us) op_driver.cc:444] REPLICATION: finished
0812 19:12:56.292162 (+ 20592us) write_op.cc:301] APPLY: starting
0812 19:12:56.292277 (+ 115us) tablet.cc:1365] starting BulkCheckPresence
0812 19:12:56.292408 (+ 131us) tablet.cc:1368] finished BulkCheckPresence
0812 19:12:56.292539 (+ 131us) tablet.cc:1370] starting ApplyRowOperation cycle
0812 19:12:56.292892 (+ 353us) tablet.cc:1381] finished ApplyRowOperation cycle
0812 19:12:56.293071 (+ 179us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0812 19:12:56.293122 (+ 51us) write_op.cc:312] APPLY: finished
0812 19:12:56.293312 (+ 190us) log.cc:842] Serialized 29 byte log entry
0812 19:12:56.293470 (+ 158us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:12:56.293562 (+ 92us) write_op.cc:454] Released schema lock
0812 19:12:56.293600 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0812 19:12:56.306848 (+ 13248us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:12:56.306884 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":20259,"apply.run_cpu_time_us":2145,"apply.run_wall_time_us":15048,"num_ops":2,"prepare.queue_time_us":4850,"prepare.run_cpu_time_us":3829,"prepare.run_wall_time_us":107500,"replication_time_us":201314,"thread_start_us":20873,"threads_started":1}]]}
I20260812 19:12:56.324527 21238 raft_consensus.cc:1261] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:56.325486 21243 raft_consensus.cc:1261] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:56.340651 22883 consensus_queue.cc:1035] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:56.363284 21033 catalog_manager.cc:5495] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 3 to 4, VOTER 93578836968649f6bfcaecc709871412 (127.19.70.131) evicted. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:56.365908 21033 catalog_manager.cc:5495] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index 3 to 4, VOTER d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) evicted. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:56.390275 21153 raft_consensus.cc:1261] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:56.395687 22900 consensus_queue.cc:1035] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:56.398722 21316 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f5090bb13ea4aa792b27b04a7a79688 with delete_type TABLET_DATA_TOMBSTONED (TS 93578836968649f6bfcaecc709871412 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51536
I20260812 19:12:56.401870 21153 raft_consensus.cc:1261] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:56.395617 21035 catalog_manager.cc:5495] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 1 to 2, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 2 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:12:56.403623 22740 consensus_queue.cc:1035] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:56.408886 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:56.410085 21022 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2f5090bb13ea4aa792b27b04a7a79688 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 19:12:56.431986 22824 consensus_queue.cc:1035] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:56.437248 21321 raft_consensus.cc:1261] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:56.439460 22901 consensus_queue.cc:1035] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:56.443346 22740 consensus_queue.cc:1035] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:56.444633 22907 tablet_replica.cc:330] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412: stopping tablet replica
I20260812 19:12:56.445866 21323 raft_consensus.cc:1261] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:56.446099 22907 raft_consensus.cc:2229] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:56.447081 21036 catalog_manager.cc:5495] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 1 to 2, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 2 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:56.451471 22907 raft_consensus.cc:2258] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:56.486491 21034 catalog_manager.cc:5495] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 1 to 2, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 2 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
W20260812 19:12:56.523094 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 55487.1us
I20260812 19:12:56.474535 22907 ts_tablet_manager.cc:1905] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:56.544677 22902 consensus_queue.cc:1035] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:56.583081 22902 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:12:56.583865 22902 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:56.588063 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:56.590528 22740 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:12:56.591408 22740 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:56.590515 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:12:56.588063 22902 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:12:56.596093 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:56.596592 21204 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:12:56.597152 22740 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:12:56.598518 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:12:56.599427 22051 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 3: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:12:56.599337 22902 raft_consensus.cc:2726] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:12:56.600574 21288 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:12:56.600621 21219 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c13f33db56241a7938cfe52798e77a2 with delete_type TABLET_DATA_TOMBSTONED (TS d1e1da0b9585460e8c365bf55679d4fa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37890
I20260812 19:12:56.602769 22740 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:56.629616 22912 tablet_replica.cc:330] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa: stopping tablet replica
I20260812 19:12:56.632618 21321 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f"
dest_uuid: "93578836968649f6bfcaecc709871412"
mode: GRACEFUL
new_leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:59192
I20260812 19:12:56.633311 21321 raft_consensus.cc:595] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Received request to transfer leadership to TS d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:56.651443 22899 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:12:56.652283 22899 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:56.658160 21321 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122"
dest_uuid: "93578836968649f6bfcaecc709871412"
mode: GRACEFUL
new_leader_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:59192
I20260812 19:12:56.658936 21321 raft_consensus.cc:595] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Received request to transfer leadership to TS f741102bfe1441d1b453e5a838649059
I20260812 19:12:56.664937 21321 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416"
dest_uuid: "93578836968649f6bfcaecc709871412"
mode: GRACEFUL
new_leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:59192
I20260812 19:12:56.665633 21321 raft_consensus.cc:595] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Received request to transfer leadership to TS d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:56.668275 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:56.669289 21237 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 3: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:12:56.672574 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:56.679879 21323 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8"
dest_uuid: "93578836968649f6bfcaecc709871412"
mode: GRACEFUL
new_leader_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:59192
I20260812 19:12:56.680753 21323 raft_consensus.cc:595] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Received request to transfer leadership to TS f741102bfe1441d1b453e5a838649059
I20260812 19:12:56.685518 22899 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:12:56.691339 22740 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:12:56.687817 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:12:56.686300 21318 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9"
dest_uuid: "93578836968649f6bfcaecc709871412"
mode: GRACEFUL
new_leader_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:59192
I20260812 19:12:56.686414 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:56.703806 21318 raft_consensus.cc:595] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Received request to transfer leadership to TS f741102bfe1441d1b453e5a838649059
I20260812 19:12:56.690459 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:12:56.696838 22893 raft_consensus.cc:979] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412: : Instructing follower d1e1da0b9585460e8c365bf55679d4fa to start an election
I20260812 19:12:56.704973 22051 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:12:56.705587 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260812 19:12:56.705946 21238 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:12:56.706568 21323 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:12:56.704937 22893 raft_consensus.cc:1067] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Signalling peer d1e1da0b9585460e8c365bf55679d4fa to start an election
I20260812 19:12:56.710556 21115 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:56.711628 22899 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:56.697353 22592 raft_consensus.cc:979] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412: : Instructing follower f741102bfe1441d1b453e5a838649059 to start an election
I20260812 19:12:56.713326 22912 raft_consensus.cc:2229] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 19:12:56.714013 22592 raft_consensus.cc:1067] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Signalling peer f741102bfe1441d1b453e5a838649059 to start an election
I20260812 19:12:56.714319 22912 raft_consensus.cc:2258] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 19:12:56.719410 21153 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:54796
I20260812 19:12:56.720814 21238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:37894
I20260812 19:12:56.721928 21238 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:56.722654 21238 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:56.723812 22912 ts_tablet_manager.cc:1905] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 19:12:56.724054 22899 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:12:56.720144 21153 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:56.726975 21153 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:56.730829 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:12:56.731853 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:56.742676 21153 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:56.745137 21153 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:56.745695 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:56.746446 21235 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:56.774657 21235 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:12:56.761842 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:56.776688 21114 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:56.777264 21323 raft_consensus.cc:3032] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:12:56.785398 22824 raft_consensus.cc:979] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412: : Instructing follower f741102bfe1441d1b453e5a838649059 to start an election
I20260812 19:12:56.780213 22897 raft_consensus.cc:2781] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:12:56.792824 22897 raft_consensus.cc:686] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:56.780063 21323 raft_consensus.cc:729] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:56.787812 22893 raft_consensus.cc:1067] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Signalling peer f741102bfe1441d1b453e5a838649059 to start an election
I20260812 19:12:56.807123 22897 consensus_queue.cc:237] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:56.799295 21238 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:56.822417 21153 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:54796
I20260812 19:12:56.823446 21153 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:56.824044 21153 raft_consensus.cc:3037] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:56.836467 22912 ts_tablet_manager.cc:1918] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:56.846477 22912 log.cc:1196] T 8c13f33db56241a7938cfe52798e77a2 P d1e1da0b9585460e8c365bf55679d4fa: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-1-root/wals/8c13f33db56241a7938cfe52798e77a2
I20260812 19:12:56.849614 21022 catalog_manager.cc:4841] TS d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): tablet 8c13f33db56241a7938cfe52798e77a2 (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:56.826030 21323 consensus_queue.cc:260] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:56.862021 21323 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:56.895900 21238 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:56.897682 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:56.898654 21321 raft_consensus.cc:3032] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:56.899078 21321 raft_consensus.cc:729] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:56.900118 21321 consensus_queue.cc:260] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:56.901706 21321 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:56.908843 21323 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:12:56.912407 21153 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:56.943009 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:56.944078 21238 raft_consensus.cc:3037] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:56.956326 21153 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:56.959362 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:56.960084 21324 raft_consensus.cc:3032] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:56.960387 21324 raft_consensus.cc:729] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:56.961145 21324 consensus_queue.cc:260] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:56.962314 21324 raft_consensus.cc:3037] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:56.965948 22705 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:56.966543 22705 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:56.969328 22705 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:12:56.975466 21324 raft_consensus.cc:2454] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:12:56.977205 21112 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:56.979598 22901 raft_consensus.cc:2781] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:56.985759 22907 ts_tablet_manager.cc:1918] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260812 19:12:56.986410 22907 log.cc:1196] T 2f5090bb13ea4aa792b27b04a7a79688 P 93578836968649f6bfcaecc709871412: Deleting WAL directory at /tmp/dist-test-task6429vE/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786561868098184-19738-0/minicluster-data/ts-2-root/wals/2f5090bb13ea4aa792b27b04a7a79688
I20260812 19:12:56.987488 22901 raft_consensus.cc:686] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:56.989055 22901 consensus_queue.cc:237] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:56.993085 21238 raft_consensus.cc:2454] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:12:56.995199 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:56.996065 21154 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:57.023041 21036 catalog_manager.cc:5495] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 2 to 3, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
W20260812 19:12:57.042366 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 236429us
I20260812 19:12:57.045114 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:12:57.054428 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:12:57.056347 21150 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:12:57.057123 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:12:57.057830 22051 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:12:57.061223 21019 catalog_manager.cc:4841] TS 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): tablet 2f5090bb13ea4aa792b27b04a7a79688 (table test-workload [id=e08e42e3e09549edabde272cb1e5858d]) successfully deleted
I20260812 19:12:57.073148 22592 raft_consensus.cc:979] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412: : Instructing follower d1e1da0b9585460e8c365bf55679d4fa to start an election
I20260812 19:12:57.073855 22812 raft_consensus.cc:1067] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Signalling peer d1e1da0b9585460e8c365bf55679d4fa to start an election
I20260812 19:12:57.046262 21318 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:12:57.078558 21154 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:12:57.080446 21200 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:57.081346 22902 raft_consensus.cc:2781] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:12:57.081807 22902 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:12:57.082121 22902 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:12:57.085035 21321 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:12:57.090507 21203 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:57.091959 22705 raft_consensus.cc:2781] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:57.092525 22705 raft_consensus.cc:686] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:57.093384 22902 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:57.093555 22705 consensus_queue.cc:237] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:57.106812 22902 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:12:57.109717 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:12:57.110661 22051 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:12:57.113883 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:57.117573 22893 raft_consensus.cc:979] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412: : Instructing follower f741102bfe1441d1b453e5a838649059 to start an election
I20260812 19:12:57.116412 21154 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:12:57.118873 21238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:37894
I20260812 19:12:57.119580 21238 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:57.120664 21238 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:57.122877 22051 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:12:57.127261 22592 raft_consensus.cc:1067] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Signalling peer f741102bfe1441d1b453e5a838649059 to start an election
I20260812 19:12:57.129204 21238 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:57.132553 21155 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:54796
I20260812 19:12:57.132704 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:57.133214 21155 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:57.133355 21150 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:57.133661 21155 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:57.141148 21155 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:57.145689 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:12:57.146021 21155 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:57.145903 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:57.147946 21318 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:12:57.148408 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:57.149049 21321 raft_consensus.cc:3032] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:57.149356 21321 raft_consensus.cc:729] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:57.146386 21235 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:57.158804 21318 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:12:57.159512 21321 consensus_queue.cc:260] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:57.160686 21200 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:12:57.163934 22705 raft_consensus.cc:2781] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:12:57.165823 22705 raft_consensus.cc:686] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:57.166801 22705 consensus_queue.cc:237] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:57.173058 21235 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:12:57.174247 21238 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:57.174141 21114 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:57.160653 21321 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:57.179893 22899 raft_consensus.cc:2781] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:57.180486 22899 raft_consensus.cc:686] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:57.181452 22899 consensus_queue.cc:237] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:57.185889 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:57.186775 21318 raft_consensus.cc:3032] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:12:57.197996 21154 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:12:57.192392 21318 raft_consensus.cc:729] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:57.206007 21150 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:12:57.208321 21321 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:12:57.205112 21318 consensus_queue.cc:260] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:57.216998 21321 raft_consensus.cc:1261] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:12:57.223714 21035 catalog_manager.cc:5495] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 1 to 2, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 2 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:57.220505 21318 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:57.231411 22897 consensus_queue.cc:1035] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:57.237099 21203 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:57.238099 22705 raft_consensus.cc:2781] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:12:57.238675 22705 raft_consensus.cc:686] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:57.239765 22705 consensus_queue.cc:237] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:57.266198 21036 catalog_manager.cc:5495] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 1 to 2, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 2 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:57.270917 21036 catalog_manager.cc:5495] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 1 to 2, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 2 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:57.282080 21318 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:12:57.327243 21036 catalog_manager.cc:5495] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:57.330832 21036 catalog_manager.cc:5495] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 2 to 3, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 3 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:57.338083 21236 raft_consensus.cc:1261] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:12:57.347271 22897 consensus_queue.cc:1035] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:57.515720 21150 raft_consensus.cc:1261] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:57.517470 22705 consensus_queue.cc:1035] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:57.550559 21318 raft_consensus.cc:1261] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:57.552624 22705 consensus_queue.cc:1035] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:57.592792 21318 raft_consensus.cc:1261] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:57.594347 22899 consensus_queue.cc:1035] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:57.598312 21318 raft_consensus.cc:1261] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:57.600020 22899 consensus_queue.cc:1035] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:57.653733 21235 raft_consensus.cc:1261] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:57.655257 21236 raft_consensus.cc:1261] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:57.660081 22899 consensus_queue.cc:1035] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:57.663640 22899 consensus_queue.cc:1035] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:12:57.668879 21155 raft_consensus.cc:1261] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:12:57.672345 22902 consensus_queue.cc:1035] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:12:57.681317 21321 raft_consensus.cc:1261] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:12:57.683497 21318 raft_consensus.cc:1261] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260812 19:12:57.685839 22705 consensus_queue.cc:1035] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:12:57.697877 22902 consensus_queue.cc:1035] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:57.723428 21324 consensus_queue.cc:237] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:57.785109 21235 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260812 19:12:57.787187 21288 consensus_peers.cc:487] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f1f5b1c2f184b59b8f452b7f8351692. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:57.792693 22051 raft_consensus.cc:1261] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260812 19:12:57.787858 22824 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:12:57.832573 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:57.833353 21150 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260812 19:12:57.837870 21150 raft_consensus.cc:1261] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260812 19:12:57.849144 22705 consensus_queue.cc:1035] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:12:57.855834 22612 consensus_queue.cc:1035] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:12:57.867059 22051 raft_consensus.cc:2932] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:12:57.838838 22740 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260812 19:12:57.945658 21235 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:12:57.946323 22740 raft_consensus.cc:2932] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:57.946367 21235 raft_consensus.cc:595] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:12:57.955384 21235 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49361681c5c1421594d10cd52061ec44"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:12:57.955960 21235 raft_consensus.cc:595] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:12:57.958501 21150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:12:57.959179 21150 raft_consensus.cc:595] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Received request to transfer leadership to TS d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:57.962764 21150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:12:57.963551 21150 raft_consensus.cc:595] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Received request to transfer leadership to TS d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:12:57.971693 22928 raft_consensus.cc:979] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059: : Instructing follower d1e1da0b9585460e8c365bf55679d4fa to start an election
I20260812 19:12:57.972266 22896 raft_consensus.cc:1067] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Signalling peer d1e1da0b9585460e8c365bf55679d4fa to start an election
I20260812 19:12:57.978605 21150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:12:57.979715 21150 raft_consensus.cc:595] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:12:57.982170 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260812 19:12:57.982826 22949 log.cc:924] Time spent T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.101s user 0.001s sys 0.000s
I20260812 19:12:57.984220 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:12:57.984983 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260812 19:12:57.989220 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 43802.7us
I20260812 19:12:57.994570 21155 raft_consensus.cc:2932] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:58.000635 21154 consensus_queue.cc:237] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:58.033690 21321 raft_consensus.cc:1261] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:58.035388 21235 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:37902
I20260812 19:12:58.036052 21235 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:58.037351 21235 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:58.040774 22897 consensus_queue.cc:1035] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260812 19:12:58.043148 21115 consensus_peers.cc:487] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c656db261bd45f8b421339ba53dd87f. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:12:58.063119 21035 catalog_manager.cc:5495] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 reported cstate change: config changed from index -1 to 3, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 2 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:58.066942 21236 raft_consensus.cc:2932] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:58.068825 21237 raft_consensus.cc:1261] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:58.084537 22897 consensus_queue.cc:1035] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:12:58.073199 21235 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:58.093451 21235 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:58.096536 22896 raft_consensus.cc:979] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059: : Instructing follower 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:12:58.097064 22896 raft_consensus.cc:1067] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Signalling peer 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:12:58.101300 21321 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7"
dest_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:51568
I20260812 19:12:58.101927 21321 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:58.102398 21321 raft_consensus.cc:3037] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:58.102945 22897 raft_consensus.cc:979] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059: : Instructing follower d1e1da0b9585460e8c365bf55679d4fa to start an election
I20260812 19:12:58.103432 22897 raft_consensus.cc:1067] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Signalling peer d1e1da0b9585460e8c365bf55679d4fa to start an election
I20260812 19:12:58.111033 21236 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:37902
I20260812 19:12:58.112025 21236 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:58.113270 21236 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:58.114367 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:58.115175 21155 raft_consensus.cc:3032] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:58.115582 21155 raft_consensus.cc:729] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:58.116412 21155 consensus_queue.cc:260] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:58.117959 21155 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:58.122824 22899 raft_consensus.cc:2932] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:58.137244 21155 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:12:58.136879 21236 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:58.140379 21236 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:58.144345 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:58.145020 21154 raft_consensus.cc:3032] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:12:58.145319 21154 raft_consensus.cc:729] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:58.145886 21203 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:58.147810 22921 raft_consensus.cc:2781] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:58.147347 21324 raft_consensus.cc:2932] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:58.148399 22921 raft_consensus.cc:686] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:12:58.155448 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 63129.2us
I20260812 19:12:58.156440 22921 consensus_queue.cc:237] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:58.158901 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:58.159621 21322 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:58.159886 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:12:58.160519 21318 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:58.146059 21154 consensus_queue.cc:260] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:58.162290 21154 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:12:58.167444 21322 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:12:58.113014 21321 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:58.174384 21321 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:12:58.162424 21236 consensus_queue.cc:237] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:12:58.178328 21154 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:12:58.181864 21203 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:12:58.190594 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:12:58.191529 22052 raft_consensus.cc:3037] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:58.192749 22705 raft_consensus.cc:2781] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:12:58.193332 22705 raft_consensus.cc:686] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:12:58.194262 22705 consensus_queue.cc:237] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:58.202071 22052 raft_consensus.cc:2454] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:58.207828 21235 raft_consensus.cc:2932] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:12:58.219564 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:12:58.220292 21155 raft_consensus.cc:3032] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:12:58.220628 21155 raft_consensus.cc:729] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:12:58.221418 21155 consensus_queue.cc:260] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:58.253197 21155 raft_consensus.cc:3037] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:58.275440 21288 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:12:58.289810 21155 raft_consensus.cc:2454] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:12:58.317683 22960 raft_consensus.cc:2781] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:12:58.393049 22960 raft_consensus.cc:686] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:12:58.420737 22960 consensus_queue.cc:237] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:58.416343 21318 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
W20260812 19:12:58.525520 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 146981us
W20260812 19:12:58.539548 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 317942us
I20260812 19:12:58.601568 21322 raft_consensus.cc:1261] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 19:12:58.583453 21034 catalog_manager.cc:5495] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260812 19:12:58.674743 21204 consensus_peers.cc:487] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18c9ea4c9e414d1093d28b6c91e19ff0. This is attempt 1: this message will repeat every 5th retry.
W20260812 19:12:58.677981 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 289493us
I20260812 19:12:58.690415 21155 raft_consensus.cc:1261] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 19:12:58.710587 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 72444.2us
W20260812 19:12:58.716426 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 80170.3us
I20260812 19:12:58.829619 22971 raft_consensus.cc:979] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa: : Instructing follower 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:12:58.830652 22971 raft_consensus.cc:1067] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Signalling peer 93578836968649f6bfcaecc709871412 to start an election
W20260812 19:12:58.848795 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 150187us
W20260812 19:12:58.893131 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 175053us
W20260812 19:12:58.904417 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 111131us
W20260812 19:12:58.970383 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 183614us
W20260812 19:12:58.949913 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 202441us
I20260812 19:12:58.946096 22612 consensus_queue.cc:1035] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260812 19:12:59.061623 21321 raft_consensus.cc:1261] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260812 19:12:59.100291 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 44896us
W20260812 19:12:59.097082 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 47235.9us
I20260812 19:12:59.105620 23006 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:12:59.145679 21322 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49361681c5c1421594d10cd52061ec44"
dest_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:51552
I20260812 19:12:59.147045 21322 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:12:59.148036 21322 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:12:59.113292 23006 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:59.185838 21156 raft_consensus.cc:1261] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:12:59.197702 21158 raft_consensus.cc:1261] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:12:59.209981 21322 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:59.230320 21324 raft_consensus.cc:1226] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:12:59.241788 21319 raft_consensus.cc:1261] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260812 19:12:59.244833 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 136371us
I20260812 19:12:59.237998 23017 ts_tablet_manager.cc:927] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:59.280887 21322 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:12:59.285846 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 155156us
I20260812 19:12:59.302722 22056 raft_consensus.cc:1261] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:12:59.318701 21034 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48428 (request call id 245) took 1138 ms. Trace:
W20260812 19:12:59.374833 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 218889us
I20260812 19:12:59.371929 23014 ts_tablet_manager.cc:927] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:59.169083 23009 consensus_queue.cc:1035] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.241s
I20260812 19:12:59.317343 22987 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:12:59.319753 21034 rpcz_store.cc:276] 0812 19:12:58.180184 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:58.180338 (+ 154us) service_pool.cc:227] Handling call
0812 19:12:59.318550 (+1138212us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0812 19:12:58.718355 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0812 19:12:58.811599 (+ 93244us) write_op.cc:432] Acquiring schema lock in shared mode
0812 19:12:58.811706 (+ 107us) write_op.cc:435] Acquired schema lock
0812 19:12:58.811762 (+ 56us) tablet.cc:660] Decoding operations
0812 19:12:58.812725 (+ 963us) write_op.cc:620] Acquiring the partition lock for write op
0812 19:12:58.812856 (+ 131us) write_op.cc:641] Partition lock acquired for write op
0812 19:12:58.812942 (+ 86us) tablet.cc:683] Acquiring locks for 1 operations
0812 19:12:58.813420 (+ 478us) tablet.cc:699] Row locks acquired
0812 19:12:58.813475 (+ 55us) write_op.cc:260] PREPARE: finished
0812 19:12:58.813883 (+ 408us) write_op.cc:270] Start()
0812 19:12:58.814137 (+ 254us) write_op.cc:276] Timestamp: P: 1786561978813755 usec, L: 0
0812 19:12:58.814187 (+ 50us) op_driver.cc:331] REPLICATION: starting
0812 19:12:58.843750 (+ 29563us) log.cc:842] Serialized 599 byte log entry
0812 19:12:58.903442 (+ 59692us) op_driver.cc:444] REPLICATION: finished
0812 19:12:58.906287 (+ 2845us) write_op.cc:301] APPLY: starting
0812 19:12:58.906470 (+ 183us) tablet.cc:1365] starting BulkCheckPresence
0812 19:12:58.906715 (+ 245us) tablet.cc:1368] finished BulkCheckPresence
0812 19:12:58.906943 (+ 228us) tablet.cc:1370] starting ApplyRowOperation cycle
0812 19:12:58.907399 (+ 456us) tablet.cc:1381] finished ApplyRowOperation cycle
0812 19:12:58.907768 (+ 369us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0812 19:12:58.907859 (+ 91us) write_op.cc:312] APPLY: finished
0812 19:12:58.908339 (+ 480us) log.cc:842] Serialized 23 byte log entry
0812 19:12:58.908730 (+ 391us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:12:58.908962 (+ 232us) write_op.cc:454] Released schema lock
0812 19:12:58.909055 (+ 93us) write_op.cc:341] FINISH: Updating metrics
0812 19:12:58.949058 (+ 40003us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:12:58.965539 (+ 16481us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":20880,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":2350,"apply.run_cpu_time_us":6892,"apply.run_wall_time_us":76919,"num_ops":1,"prepare.queue_time_us":9358,"prepare.run_cpu_time_us":7410,"prepare.run_wall_time_us":170419,"replication_time_us":89036}]]}
W20260812 19:12:59.497759 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 164853us
I20260812 19:12:59.327004 22705 consensus_queue.cc:1035] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.146s
I20260812 19:12:59.422155 23006 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:12:59.369590 22612 consensus_queue.cc:1035] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.105s
W20260812 19:12:59.551126 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 271361us
I20260812 19:12:59.528971 23023 ts_tablet_manager.cc:927] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551)
W20260812 19:12:59.556041 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 240317us
I20260812 19:12:59.513269 22972 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:12:59.574396 22978 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:12:59.570034 22740 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:12:59.484858 22987 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:59.545395 22988 consensus_queue.cc:1035] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260812 19:12:59.588636 22972 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:59.616182 22967 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:12:59.724179 23028 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:12:59.633411 22978 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:59.618119 23005 consensus_queue.cc:1035] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.124s
I20260812 19:12:59.746249 23028 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:12:59.781116 22972 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:12:59.785211 23028 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:12:59.740267 22967 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:59.796308 22987 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:59.807017 23014 tablet_copy_client.cc:316] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.130:33807
I20260812 19:12:59.877676 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:12:59.883852 21035 catalog_manager.cc:5495] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 1 to 2, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 2 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:59.816847 23017 tablet_copy_client.cc:316] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.131:33285
I20260812 19:12:59.813856 23023 tablet_copy_client.cc:316] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.129:36551
I20260812 19:12:59.631024 22740 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:12:59.934981 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 460378us
I20260812 19:12:59.963052 23019 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:12:59.941461 21035 catalog_manager.cc:5495] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 2 to 3, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 3 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:12:59.970484 23022 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:12:59.972707 23005 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:12:59.972782 23022 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:59.993352 23005 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:12:59.999517 23043 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:00.010361 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 347395us
I20260812 19:13:00.001938 23043 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:00.021335 22978 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:00.008065 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 401618us
I20260812 19:13:00.031788 22960 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:00.042557 22972 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:00.041846 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:00.049194 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:00.057017 21153 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:00.045045 22972 raft_consensus.cc:504] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:00.028127 22740 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:00.033916 22960 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:00.071271 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 515577us
W20260812 19:13:00.071599 21286 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.011s sys 0.008s
W20260812 19:13:00.076989 21286 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.722s user 0.012s sys 0.008s
I20260812 19:13:00.078455 23005 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:00.078804 23022 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:12:59.966037 23019 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:00.023963 23049 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:00.096998 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:00.098328 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:00.083688 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:00.076784 23043 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:00.082594 22972 leader_election.cc:288] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:00.126255 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:00.126731 21169 tablet_copy_service.cc:140] P f741102bfe1441d1b453e5a838649059: Received BeginTabletCopySession request for tablet 4c656db261bd45f8b421339ba53dd87f from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:51004)
I20260812 19:13:00.128430 21169 tablet_copy_service.cc:161] P f741102bfe1441d1b453e5a838649059: Beginning new tablet copy session on tablet 4c656db261bd45f8b421339ba53dd87f from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:51004: session id = 63db10063a7d4eb2a9cc9dc952654706-4c656db261bd45f8b421339ba53dd87f
I20260812 19:13:00.133145 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:00.076907 22967 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:00.160372 23024 consensus_queue.cc:1035] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260812 19:13:00.166136 22983 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:00.167949 22983 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:00.231113 23019 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:00.235225 22967 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:00.236166 22972 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:00.237196 22967 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:00.242115 21150 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:13:00.237536 22972 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:00.091619 23049 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:00.315342 22896 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:00.359721 22896 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:00.395269 23049 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:00.407078 22960 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:00.433028 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:00.422853 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:00.302241 23019 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:00.447289 22960 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:00.416425 22997 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:00.446484 23019 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:00.453800 22896 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:00.449517 22960 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:00.468238 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:00.412519 22972 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:00.471276 23048 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:00.475844 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 264927us
I20260812 19:13:00.483358 23037 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:00.420101 22967 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:00.452333 22997 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:00.440234 22983 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:00.497123 23048 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:00.501722 22976 consensus_queue.cc:1046] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260812 19:13:00.512780 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:00.529778 22967 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:00.464627 23049 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:00.502939 23037 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:00.503239 22963 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:00.503446 23024 raft_consensus.cc:2932] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:13:00.552328 23010 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:00.567770 22967 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:00.593991 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:00.633723 22716 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:00.635282 23019 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:00.635542 22716 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:00.635502 21342 tablet_copy_service.cc:140] P 93578836968649f6bfcaecc709871412: Received BeginTabletCopySession request for tablet 6f1f5b1c2f184b59b8f452b7f8351692 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:50042)
I20260812 19:13:00.619963 22960 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:00.666361 21342 tablet_copy_service.cc:161] P 93578836968649f6bfcaecc709871412: Beginning new tablet copy session on tablet 6f1f5b1c2f184b59b8f452b7f8351692 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:50042: session id = 63db10063a7d4eb2a9cc9dc952654706-6f1f5b1c2f184b59b8f452b7f8351692
I20260812 19:13:00.620754 22978 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:00.687342 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:00.642221 23049 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:00.695725 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:00.598163 22963 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:00.669864 21254 tablet_copy_service.cc:140] P d1e1da0b9585460e8c365bf55679d4fa: Received BeginTabletCopySession request for tablet 18c9ea4c9e414d1093d28b6c91e19ff0 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:57682)
I20260812 19:13:00.624549 23010 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:00.698436 21036 catalog_manager.cc:5495] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 1 to 2, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 2 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:13:00.710188 22997 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:00.787322 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:00.790292 21254 tablet_copy_service.cc:161] P d1e1da0b9585460e8c365bf55679d4fa: Beginning new tablet copy session on tablet 18c9ea4c9e414d1093d28b6c91e19ff0 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:57682: session id = 63db10063a7d4eb2a9cc9dc952654706-18c9ea4c9e414d1093d28b6c91e19ff0
I20260812 19:13:00.758251 23007 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:00.791890 23048 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:00.793007 22740 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:00.794231 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:00.742502 23019 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:00.791675 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:00.735944 22972 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:00.827873 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:00.751395 22983 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:00.799054 23043 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:00.837178 21153 consensus_meta.cc:287] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.779s user 0.018s sys 0.011s
I20260812 19:13:00.845424 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:00.798416 22960 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:00.856529 21153 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:00.824862 23037 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:00.809525 22978 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:00.885828 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:00.866688 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:00.884593 23048 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:00.866259 22740 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:00.936707 22716 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:00.874354 22983 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:00.939189 21154 raft_consensus.cc:1226] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:00.903430 23037 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:00.897912 22967 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:00.876106 23019 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:00.942262 23037 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:00.947647 23019 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:00.948463 22963 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:00.933106 21169 tablet_copy_source_session.cc:205] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:13:00.905332 23010 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:00.870357 23007 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:00.881743 23043 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:00.892758 22960 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:00.973181 23010 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:00.976245 22978 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:00.950021 22963 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:00.978324 23048 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:00.979698 22963 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:00.984046 22740 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:00.985364 22963 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:00.985802 22740 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:00.987748 23037 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:00.986867 23010 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
W20260812 19:13:01.014398 21200 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.010s sys 0.032s
W20260812 19:13:01.015362 21200 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.513s user 0.010s sys 0.032s
I20260812 19:13:00.915602 22972 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:01.083377 21202 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.043s sys 0.041s
I20260812 19:13:01.053771 22983 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:01.086165 21202 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.556s user 0.044s sys 0.041s
I20260812 19:13:01.064584 23049 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:01.080106 23048 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:01.036240 22740 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:01.132576 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 194339us
W20260812 19:13:01.186669 21158 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.008s sys 0.010s
W20260812 19:13:01.194295 21158 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.770s user 0.008s sys 0.012s
I20260812 19:13:01.169251 22896 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
W20260812 19:13:01.196549 21203 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.001s sys 0.048s
W20260812 19:13:01.205986 21203 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.694s user 0.002s sys 0.048s
I20260812 19:13:01.183883 23059 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:01.224409 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:01.185992 23043 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:01.232380 23043 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:01.183310 22967 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:01.210229 22960 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:01.216697 23007 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:01.283854 22740 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:01.284212 22612 raft_consensus.cc:3032] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:01.285089 22612 raft_consensus.cc:729] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:01.287001 23057 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:01.273643 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:01.290562 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 131563us
I20260812 19:13:01.299520 23073 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
W20260812 19:13:01.302815 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 376766us
W20260812 19:13:01.304219 23052 log.cc:924] Time spent T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.236s user 0.000s sys 0.003s
W20260812 19:13:01.307873 23054 log.cc:924] Time spent T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.119s user 0.000s sys 0.002s
I20260812 19:13:01.309376 21156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1554) took 1196 ms. Trace:
I20260812 19:13:01.308323 22896 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:01.310864 22963 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:01.311755 23078 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:01.312433 22963 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:01.295195 21323 raft_consensus.cc:2454] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:13:01.264959 23010 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:01.313110 23078 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:01.233937 23043 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:01.317705 23010 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:01.331573 23058 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:01.332559 23010 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:01.334564 22976 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:01.336009 22976 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:01.338971 23064 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:01.288880 23057 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:01.368314 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 166244us
I20260812 19:13:01.301168 23073 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:01.266883 23022 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:01.338220 23010 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:01.302336 22716 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:01.274135 22960 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:01.416108 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:01.419502 23074 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:01.421298 23070 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:01.419255 21115 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:01.345007 23048 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:01.375938 22612 consensus_queue.cc:260] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:01.310340 21156 rpcz_store.cc:276] 0812 19:13:00.115357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:00.119345 (+ 3988us) service_pool.cc:227] Handling call
0812 19:13:00.138319 (+ 18974us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:00.139505 (+ 1186us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:00.139566 (+ 61us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:00.141001 (+ 1435us) log.cc:842] Serialized 26 byte log entry
0812 19:13:00.600422 (+459421us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:00.600495 (+ 73us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:00.600619 (+ 124us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:01.308425 (+707806us) raft_consensus.cc:1671] finished
0812 19:13:01.308487 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:01.308830 (+ 343us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3407,"raft.run_cpu_time_us":426,"raft.run_wall_time_us":21039,"thread_start_us":177917,"threads_started":1,"wal-append.queue_time_us":785454}
I20260812 19:13:01.477239 23092 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:01.357513 22967 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:01.494143 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 75026us
I20260812 19:13:01.496031 21115 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:01.334679 23058 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:01.497851 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:01.372820 23059 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:01.512066 23071 raft_consensus.cc:2726] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:01.472470 23043 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:01.521560 22612 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
W20260812 19:13:01.488281 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 153347us
W20260812 19:13:01.532477 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 104043us
I20260812 19:13:01.534397 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:01.535264 23062 raft_consensus.cc:2726] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:01.536710 22967 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:01.493332 22972 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:01.460480 22960 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:01.340250 23064 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:01.555760 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:01.553277 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:01.479420 23092 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:01.500914 23048 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:01.489168 23022 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:01.568370 22896 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:01.524863 23070 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:01.578974 23085 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:01.527962 23037 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:01.538293 23057 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:01.606449 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 43850.5us
I20260812 19:13:01.606667 21112 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:01.489797 22716 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:01.634202 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 315475us
W20260812 19:13:01.636917 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 249690us
I20260812 19:13:01.560985 22997 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:01.645226 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:01.648882 23061 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:01.628012 23059 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:01.660790 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 62409.7us
I20260812 19:13:01.500571 23074 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:01.641274 23010 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:01.676461 21342 tablet_copy_source_session.cc:205] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:13:01.677512 23010 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:01.641996 23078 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:01.683465 22716 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:01.690666 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:01.637928 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 70972us
I20260812 19:13:01.655864 22896 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:01.699138 21342 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.BeginTabletCopySession from 127.0.0.1:50042 (request call id 177) took 1110 ms. Trace:
I20260812 19:13:01.698959 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:01.700038 21342 rpcz_store.cc:276] 0812 19:13:00.588453 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:00.634827 (+ 46374us) service_pool.cc:227] Handling call
0812 19:13:01.698888 (+1064061us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:01.701748 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:01.702519 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:01.702853 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:01.676530 22960 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:01.704160 21114 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 1.623s (SENT) [suppressed 4 similar messages]
I20260812 19:13:01.704586 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:01.649029 23088 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:01.653926 22976 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:01.691699 23055 log.cc:924] Time spent T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.158s user 0.000s sys 0.001s
I20260812 19:13:01.667052 23071 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:01.693950 23010 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:01.706095 21149 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:01.692597 23062 raft_consensus.cc:2781] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:01.723713 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:01.693889 23037 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:01.696117 22963 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:01.721091 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:01.729671 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:01.719480 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:01.737551 21150 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:01.732448 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:01.732774 21152 raft_consensus.cc:1261] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:13:01.733168 23048 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:01.733685 23092 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:01.735389 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:01.731129 21153 raft_consensus.cc:2454] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:13:01.745110 21158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1558) took 1507 ms. Trace:
I20260812 19:13:01.672119 23085 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:01.798724 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:01.707320 22997 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:01.803800 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:01.742340 23064 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:01.771760 21158 rpcz_store.cc:276] 0812 19:13:00.238439 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:00.238820 (+ 381us) service_pool.cc:227] Handling call
0812 19:13:00.239781 (+ 961us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:00.241096 (+ 1315us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:00.241185 (+ 89us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:00.242501 (+ 1316us) log.cc:842] Serialized 26 byte log entry
0812 19:13:01.203060 (+960559us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:01.203160 (+ 100us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:01.203294 (+ 134us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:01.744672 (+541378us) raft_consensus.cc:1671] finished
0812 19:13:01.744710 (+ 38us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:01.744977 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1022,"raft.run_cpu_time_us":507,"raft.run_wall_time_us":504,"thread_start_us":770716,"threads_started":1,"wal-append.queue_time_us":912891}
I20260812 19:13:01.736974 23062 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:01.742498 23037 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:01.745294 23077 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:01.813655 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:01.746132 23070 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:01.772130 22978 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:01.804172 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:01.817593 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 165916us
I20260812 19:13:01.719277 22896 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:01.758270 23022 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:01.840906 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 136839us
I20260812 19:13:01.835640 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:01.846130 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:01.849592 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:01.808174 23073 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:01.822134 23062 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:01.850885 21153 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:13:01.744465 23071 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:01.825971 23058 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:01.885848 21154 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:01.854990 23090 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:01.808787 23088 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:01.696820 23061 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:01.867229 22978 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:02.544773 22612 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:01.922735 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:02.040285 23071 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:02.115672 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 464729us
I20260812 19:13:02.107636 23023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c656db261bd45f8b421339ba53dd87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:13:02.134940 23075 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:02.560256 23075 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:01.927896 23082 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:02.030061 23073 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:02.564481 23082 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:02.030522 22896 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:02.100132 23076 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
W20260812 19:13:02.319824 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 257582us
W20260812 19:13:01.923803 23063 log.cc:924] Time spent T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1: Append to log took a long time: real 0.086s user 0.001s sys 0.001s
I20260812 19:13:01.923019 23093 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
W20260812 19:13:02.116688 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 453752us
I20260812 19:13:02.557169 22612 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:02.020917 23043 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:02.448396 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:01.906946 23024 consensus_meta.cc:287] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.041s user 0.030s sys 0.000s
I20260812 19:13:02.453469 21035 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48434 (request call id 249) took 3851 ms. Trace:
I20260812 19:13:02.614178 22612 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:01.887946 23072 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:02.605548 21240 consensus_meta.cc:287] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.736s user 0.019s sys 0.002s
I20260812 19:13:02.617264 21240 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:02.617549 23090 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:02.607887 23073 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:02.620560 23043 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:02.623319 23073 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:02.617908 23093 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:02.625412 23043 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:02.608785 22051 raft_consensus.cc:2454] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:13:02.629644 23093 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:02.631191 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:02.011165 23065 log.cc:924] Time spent T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
I20260812 19:13:02.569973 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:02.635821 21288 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
W20260812 19:13:02.640887 22011 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.776s user 0.029s sys 0.022s
W20260812 19:13:02.643260 22011 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.779s user 0.029s sys 0.023s
I20260812 19:13:01.903823 23094 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:02.645787 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:02.647411 21233 raft_consensus.cc:2454] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:02.221916 22740 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:02.040146 22997 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:02.055442 23049 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:02.640478 22896 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:02.622601 23090 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:01.888785 23077 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:02.136178 23074 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:02.552560 23085 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:02.650187 21287 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:02.631251 23061 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:02.698199 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:02.707124 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:02.730988 22740 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:02.751876 23024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:57672 (request call id 11) took 4599 ms. Trace:
I20260812 19:13:02.752879 23024 rpcz_store.cc:276] 0812 19:12:58.152378 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:58.160259 (+ 7881us) service_pool.cc:227] Handling call
0812 19:12:58.210043 (+ 49784us) log.cc:842] Serialized 506 byte log entry
0812 19:12:58.457782 (+247739us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b54001d1620. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9a37a4f 00007f52d9a20545 00007f52d9a25771 00007f52d9a256f9 00007f52d9a25689 00007f52d9a25651 00007f52d9a2481d 00007f52dfde2204 00007f52dfde2039 00007f52d6b8a6b6
0812 19:12:58.461549 (+ 3767us) spinlock_profiling.cc:243] Waited 1.53 ms on lock 0x7b54001d1620. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9a37a4f 00007f52d9a20545 00007f52d9a25771 00007f52d9a256f9 00007f52d9a25689 00007f52d9a25651 00007f52d9a2481d 00007f52dfde2204 00007f52dfde2039 00007f52d6b8a6b6
0812 19:13:02.751618 (+4290069us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8657,"raft.queue_time_us":151612,"raft.run_cpu_time_us":9734,"raft.run_wall_time_us":187733,"spinlock_wait_cycles":45091072,"thread_start_us":214856,"threads_started":3,"wal-append.queue_time_us":73188,"wal-append.run_cpu_time_us":4280,"wal-append.run_wall_time_us":1020611}
I20260812 19:13:02.739500 22896 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:02.616247 21035 rpcz_store.cc:276] 0812 19:12:58.585378 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:58.585708 (+ 330us) service_pool.cc:227] Handling call
0812 19:13:02.436723 (+3851015us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0812 19:13:00.437695 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0812 19:13:00.438912 (+ 1217us) write_op.cc:432] Acquiring schema lock in shared mode
0812 19:13:00.438975 (+ 63us) write_op.cc:435] Acquired schema lock
0812 19:13:00.439026 (+ 51us) tablet.cc:660] Decoding operations
0812 19:13:00.440132 (+ 1106us) write_op.cc:620] Acquiring the partition lock for write op
0812 19:13:00.474052 (+ 33920us) write_op.cc:641] Partition lock acquired for write op
0812 19:13:00.493708 (+ 19656us) tablet.cc:683] Acquiring locks for 2 operations
0812 19:13:00.647122 (+153414us) tablet.cc:699] Row locks acquired
0812 19:13:00.660653 (+ 13531us) write_op.cc:260] PREPARE: finished
0812 19:13:00.756204 (+ 95551us) write_op.cc:270] Start()
0812 19:13:00.786915 (+ 30711us) write_op.cc:276] Timestamp: P: 1786561980739976 usec, L: 0
0812 19:13:00.787026 (+ 111us) op_driver.cc:331] REPLICATION: starting
0812 19:13:00.789043 (+ 2017us) log.cc:842] Serialized 929 byte log entry
0812 19:13:01.753607 (+964564us) op_driver.cc:444] REPLICATION: finished
0812 19:13:01.840622 (+ 87015us) write_op.cc:301] APPLY: starting
0812 19:13:01.867989 (+ 27367us) tablet.cc:1365] starting BulkCheckPresence
0812 19:13:02.012187 (+144198us) tablet.cc:1368] finished BulkCheckPresence
0812 19:13:02.036623 (+ 24436us) tablet.cc:1370] starting ApplyRowOperation cycle
0812 19:13:02.051570 (+ 14947us) tablet.cc:1381] finished ApplyRowOperation cycle
0812 19:13:02.052196 (+ 626us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0812 19:13:02.052342 (+ 146us) write_op.cc:312] APPLY: finished
0812 19:13:02.053123 (+ 781us) log.cc:842] Serialized 29 byte log entry
0812 19:13:02.053481 (+ 358us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:13:02.053765 (+ 284us) write_op.cc:454] Released schema lock
0812 19:13:02.053969 (+ 204us) write_op.cc:341] FINISH: Updating metrics
0812 19:13:02.086278 (+ 32309us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:13:02.092499 (+ 6221us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":58265,"apply.run_cpu_time_us":7412,"apply.run_wall_time_us":296221,"num_ops":2,"prepare.queue_time_us":305428,"prepare.run_cpu_time_us":31242,"prepare.run_wall_time_us":676257,"raft.queue_time_us":277672,"raft.run_cpu_time_us":12171,"raft.run_wall_time_us":89632,"replication_time_us":966338,"thread_start_us":781955,"threads_started":5,"wal-append.queue_time_us":402913}]]}
I20260812 19:13:02.712634 22997 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:02.741139 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 132866us
I20260812 19:13:02.640579 23076 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:02.762276 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:02.765260 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:02.773690 22053 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 4.
I20260812 19:13:02.766286 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:02.761844 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:02.766901 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:02.765908 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:02.774807 21154 raft_consensus.cc:2454] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:02.711005 23087 raft_consensus.cc:2781] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:02.761188 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1534) took 2231 ms. Trace:
I20260812 19:13:02.767923 22997 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:02.768900 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:02.793792 21244 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
I20260812 19:13:02.780440 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1394) took 1119 ms. Trace:
I20260812 19:13:02.794633 21240 rpcz_store.cc:276] 0812 19:13:01.686105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:01.693529 (+ 7424us) service_pool.cc:227] Handling call
0812 19:13:02.780224 (+1086695us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b54003bf454. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:02.780341 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":356314368}
I20260812 19:13:02.795225 22896 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:02.754565 22740 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:02.796052 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:02.796803 21240 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:02.797729 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:02.781337 21287 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:02.783084 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:02.798261 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:02.781126 22983 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:02.799008 21237 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:13:02.777345 22055 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:02.767395 23075 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:02.705329 23104 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:02.780709 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:02.800297 23104 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:02.801378 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:02.731599 23094 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:02.733047 23049 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:02.799312 22983 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:02.760836 23062 consensus_meta.cc:287] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.872s user 0.015s sys 0.007s
I20260812 19:13:02.796172 23082 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:02.809938 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:02.761406 23074 raft_consensus.cc:2781] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260812 19:13:02.791332 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 90609.8us
I20260812 19:13:02.807770 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:02.815496 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:02.789584 23087 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:02.817898 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:02.845134 21244 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:02.713560 23072 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:02.813676 23069 raft_consensus.cc:2781] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:13:02.799969 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:02.808951 21203 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:02.845938 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:02.814471 23062 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:02.813681 22896 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:02.844724 23071 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:02.853261 23088 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:02.854161 23094 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:02.823238 23074 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:02.866361 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 57436.9us
I20260812 19:13:02.846876 23087 raft_consensus.cc:3037] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:02.844498 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:02.894312 21153 raft_consensus.cc:2379] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 2: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:02.872473 23071 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:02.873021 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:02.855026 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:02.902024 23117 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:02.868944 23069 raft_consensus.cc:686] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:02.912740 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:02.913455 21319 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:02.913540 23069 consensus_queue.cc:237] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:02.916132 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:02.918601 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:02.918902 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:02.921342 23071 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:02.870795 23024 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:02.891584 22896 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:02.870925 23085 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:02.894284 23074 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:02.898314 23076 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:02.898794 23103 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:02.789680 21318 rpcz_store.cc:276] 0812 19:13:00.530039 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:00.542677 (+ 12638us) service_pool.cc:227] Handling call
0812 19:13:00.630702 (+ 88025us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:00.771256 (+140554us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:00.774258 (+ 3002us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:00.807790 (+ 33532us) log.cc:842] Serialized 26 byte log entry
0812 19:13:01.112077 (+304287us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:01.124786 (+ 12709us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:01.149504 (+ 24718us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:02.760908 (+1611404us) raft_consensus.cc:1671] finished
0812 19:13:02.760939 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:02.761095 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55628,"raft.queue_time_us":92467,"raft.run_cpu_time_us":366,"raft.run_wall_time_us":16721,"thread_start_us":292046,"threads_started":1,"wal-append.queue_time_us":865451}
I20260812 19:13:02.850688 23111 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:02.891121 23116 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:02.902745 23117 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:02.904940 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37894 (request call id 1400) took 1155 ms. Trace:
I20260812 19:13:02.905051 23072 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:02.905720 21200 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:02.909759 23049 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:02.910646 21203 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:03.045465 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:02.911021 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:02.912220 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:03.046558 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 136081us
I20260812 19:13:03.047277 22612 raft_consensus.cc:2781] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:02.923086 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:02.923182 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:03.047706 22612 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:02.928440 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:02.891108 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:02.993930 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 177572us
I20260812 19:13:03.046942 21324 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:03.017906 22997 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:03.015230 23085 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:03.017066 22983 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:03.019078 22740 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:03.060920 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.020265 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:03.027482 23110 log.cc:924] Time spent T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20260812 19:13:03.028326 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.063119 23085 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:03.063310 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.063992 22740 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:02.993762 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 147065us
I20260812 19:13:03.065471 22896 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:03.053273 23088 raft_consensus.cc:2726] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:03.067165 23049 consensus_queue.cc:1035] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:13:03.067502 21114 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:02.982107 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:03.033105 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:03.038457 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 132810us
I20260812 19:13:03.042888 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.069706 21320 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:03.044168 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.037563 21243 rpcz_store.cc:276] 0812 19:13:01.749227 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:01.752099 (+ 2872us) service_pool.cc:227] Handling call
0812 19:13:01.752748 (+ 649us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:02.644588 (+891840us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:02.644646 (+ 58us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:02.644737 (+ 91us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:02.644794 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:02.904587 (+259793us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b54001518d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:02.904640 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:02.904744 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2021792640}
I20260812 19:13:02.912431 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.045490 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.048117 22055 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:13:03.048277 22612 raft_consensus.cc:3037] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:02.896803 23104 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:03.057665 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.059250 23062 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:03.059499 21254 tablet_copy_source_session.cc:205] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:13:03.059386 23094 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:03.085209 23094 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:03.087451 23094 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:03.060102 23075 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:03.088292 23075 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:03.088954 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.089924 21319 raft_consensus.cc:2454] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:03.074095 22011 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:03.066489 21287 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:02.927985 23024 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:03.073038 21154 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
I20260812 19:13:03.070191 21150 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
I20260812 19:13:03.086164 22978 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:03.079243 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.079453 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.080585 21151 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
I20260812 19:13:03.081667 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.084299 23117 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:03.035413 23111 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:03.079067 21114 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:03.099370 21235 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1295) took 1403 ms. Trace:
I20260812 19:13:03.096163 21327 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:03.103259 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.103730 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.104297 21322 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:03.105593 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.107116 23127 raft_consensus.cc:2726] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:03.107425 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.108103 21320 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:03.110086 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.110782 23062 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:03.107005 22997 raft_consensus.cc:2726] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:03.111042 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.111583 22997 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:03.111382 23062 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:03.111931 21318 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:03.112159 22997 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:03.113238 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.113641 23062 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:03.099997 21235 rpcz_store.cc:276] 0812 19:13:01.696531 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:01.696786 (+ 255us) service_pool.cc:227] Handling call
0812 19:13:01.698908 (+ 2122us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:02.622950 (+924042us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:02.623005 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:02.623165 (+ 160us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:02.623232 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:03.098994 (+475762us) spinlock_profiling.cc:243] Waited 924 ms on lock 0x7b54002567d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:03.099079 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:03.099275 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2124312320}
I20260812 19:13:03.116616 22011 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:03.106894 23123 raft_consensus.cc:2781] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260812 19:13:03.118028 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 51633.7us
I20260812 19:13:03.118286 23123 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:03.118821 23123 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:03.119624 21287 leader_election.cc:302] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:03.122010 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.123988 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.066500 22740 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:03.128263 22612 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:03.095326 23024 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:03.130658 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.095494 23075 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:03.126776 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.098416 23094 raft_consensus.cc:2781] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:03.133855 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.134111 21114 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:03.131927 22011 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:03.135056 21287 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:03.136471 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.136320 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.135394 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.114315 22997 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:03.138005 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.138093 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.138576 22612 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:03.137202 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.054673 23103 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:03.139227 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:03.407261 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 272271us
I20260812 19:13:03.406466 21153 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:03.247160 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:03.408732 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 274705us
I20260812 19:13:03.409940 21114 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:03.205333 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.411509 21156 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:03.208253 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.148164 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.413293 21149 raft_consensus.cc:2454] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:03.205783 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.410928 21152 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:13:03.127111 23117 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:03.132601 23104 raft_consensus.cc:2781] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:03.130578 23062 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:03.434690 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.435071 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.434930 23062 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:03.435496 21322 raft_consensus.cc:2454] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:03.436378 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.437119 21328 raft_consensus.cc:2454] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:13:03.437559 23062 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:03.437718 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.276373 23077 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:03.438874 23111 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:03.259473 23087 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:03.440027 21324 raft_consensus.cc:1226] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:03.235318 21254 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.BeginTabletCopySession from 127.0.0.1:57682 (request call id 115) took 2673 ms. Trace:
I20260812 19:13:03.428962 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.134678 23105 raft_consensus.cc:2726] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:03.437873 23104 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:03.440974 21254 rpcz_store.cc:276] 0812 19:13:00.596361 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:00.669232 (+ 72871us) service_pool.cc:227] Handling call
0812 19:13:03.059076 (+2389844us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b5400043dd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9ab0257 00007f52de9c35e6 00007f52de9cce5e 00007f52d64b35db 00007f52de9ccdda 00007f52de9c7eec 00007f52de9c324b 00007f52de9b7e45 00007f52ddac0a54 00007f52ddac09e2
0812 19:13:03.235196 (+176120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3948786560}
I20260812 19:13:03.207715 23109 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:03.433665 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.447131 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.447840 21319 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:03.244678 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.451581 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.447849 21114 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:03.458710 21321 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:03.459416 21114 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:03.417615 23120 raft_consensus.cc:2726] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:03.417985 23121 raft_consensus.cc:2726] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:03.461102 21155 raft_consensus.cc:1226] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
W20260812 19:13:03.467054 21287 connection.cc:232] RPC call timeout handler was delayed by -2.5033747979905456s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1506s for a response
I20260812 19:13:03.460389 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.450634 22612 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:03.476933 22612 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:03.420476 23075 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:03.415831 23094 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:03.417377 23049 raft_consensus.cc:2781] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:03.406479 21150 raft_consensus.cc:2932] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
W20260812 19:13:03.477459 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 345577us
I20260812 19:13:03.480422 22612 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:03.473189 21318 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
W20260812 19:13:03.492465 21287 connection.cc:232] RPC call timeout handler was delayed by -2.1352747980272397s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1915s for a response
I20260812 19:13:03.447136 23104 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:03.128057 23116 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:03.507578 22011 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:03.513785 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.454507 21323 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:13:03.515437 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.515774 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.515340 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.516739 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.469957 22997 raft_consensus.cc:2726] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 19:13:03.521250 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 77860.5us
I20260812 19:13:03.520077 22011 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:03.456342 23087 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:03.526291 21235 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:13:03.525468 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.530110 21321 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:03.517827 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.524101 21243 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:03.527170 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.531836 21233 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:13:03.473695 23127 raft_consensus.cc:2781] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260812 19:13:03.533107 21287 connection.cc:232] RPC call timeout handler was delayed by -1.211140566971153s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1572s for a response
W20260812 19:13:03.533563 21287 connection.cc:232] RPC call timeout handler was delayed by -1.1202405670192093s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1673s for a response
W20260812 19:13:03.533906 21287 connection.cc:232] RPC call timeout handler was delayed by -1.0068405669881031s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1799s for a response
I20260812 19:13:03.535495 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.541492 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.536150 21320 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:03.497619 23094 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:03.448066 23117 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:03.547001 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.549695 21233 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:03.544420 21158 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
I20260812 19:13:03.494822 22983 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:03.494586 23049 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:03.492121 23075 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:03.525997 23125 log.cc:924] Time spent T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.120s user 0.000s sys 0.001s
I20260812 19:13:03.556257 21036 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58998 (request call id 139) took 2888 ms. Trace:
I20260812 19:13:03.559001 22011 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:03.518538 23111 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:03.552711 21158 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51004 (request call id 138) took 1673 ms (client timeout 2217 ms). Trace:
I20260812 19:13:03.454764 23109 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:03.528916 23082 raft_consensus.cc:2781] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:03.566375 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.522585 23121 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:03.536690 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.600967 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:03.599149 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.606047 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.502465 23074 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
W20260812 19:13:03.568262 21158 rpcz_store.cc:269] 0812 19:13:01.879339 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:02.910708 (+1031369us) service_pool.cc:227] Handling call
0812 19:13:03.552481 (+641773us) spinlock_profiling.cc:243] Waited 495 ms on lock 0x7b5400194bd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:03.552643 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1138960640}
I20260812 19:13:03.540772 23127 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:03.547663 23069 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:03.547346 23132 raft_consensus.cc:2781] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:03.640316 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.635084 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.563608 23049 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:03.654011 23017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1f5b1c2f184b59b8f452b7f8351692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:13:03.654479 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:03.656023 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.656744 21156 raft_consensus.cc:2454] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:13:03.564296 23075 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
W20260812 19:13:03.640056 23123 consensus_meta.cc:287] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.521s user 0.013s sys 0.002s
I20260812 19:13:03.567968 23093 raft_consensus.cc:2726] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:03.645908 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:03.646860 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.635672 23121 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:03.597765 23117 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:03.598731 23023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:40624 (request call id 265) took 4360 ms. Trace:
I20260812 19:13:03.603155 23116 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:03.621802 21240 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:03.658442 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:03.663707 21329 raft_consensus.cc:1226] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:03.601984 23103 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:03.666141 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:03.636766 23082 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:03.692516 23082 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:03.693208 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.695058 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.695910 21244 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:03.697079 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.697837 21244 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:03.568262 22983 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:03.669160 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.670151 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.671090 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.672194 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.676174 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.678159 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.642561 23127 raft_consensus.cc:3037] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:03.668069 21153 consensus_queue.cc:237] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:03.644795 23132 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:03.684263 22011 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:03.565160 21036 rpcz_store.cc:276] 0812 19:13:00.667746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:00.668308 (+ 562us) service_pool.cc:227] Handling call
0812 19:13:03.556143 (+2887835us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0812 19:13:01.200462 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0812 19:13:01.236420 (+ 35958us) write_op.cc:432] Acquiring schema lock in shared mode
0812 19:13:01.236555 (+ 135us) write_op.cc:435] Acquired schema lock
0812 19:13:01.236595 (+ 40us) tablet.cc:660] Decoding operations
0812 19:13:01.237355 (+ 760us) write_op.cc:620] Acquiring the partition lock for write op
0812 19:13:01.237474 (+ 119us) write_op.cc:641] Partition lock acquired for write op
0812 19:13:01.237566 (+ 92us) tablet.cc:683] Acquiring locks for 1 operations
0812 19:13:01.238106 (+ 540us) tablet.cc:699] Row locks acquired
0812 19:13:01.238141 (+ 35us) write_op.cc:260] PREPARE: finished
0812 19:13:01.238564 (+ 423us) write_op.cc:270] Start()
0812 19:13:01.265357 (+ 26793us) write_op.cc:276] Timestamp: P: 1786561981238405 usec, L: 0
0812 19:13:01.265433 (+ 76us) op_driver.cc:331] REPLICATION: starting
0812 19:13:01.315944 (+ 50511us) log.cc:842] Serialized 536 byte log entry
0812 19:13:02.917638 (+1601694us) op_driver.cc:444] REPLICATION: finished
0812 19:13:03.270689 (+353051us) write_op.cc:301] APPLY: starting
0812 19:13:03.277745 (+ 7056us) tablet.cc:1365] starting BulkCheckPresence
0812 19:13:03.338564 (+ 60819us) tablet.cc:1368] finished BulkCheckPresence
0812 19:13:03.362517 (+ 23953us) tablet.cc:1370] starting ApplyRowOperation cycle
0812 19:13:03.456247 (+ 93730us) spinlock_profiling.cc:243] Waited 1.42 ms on lock 0x7b5c00096b8c. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfe2f501 00007f52dfe2f3f1 00007f52dfe2a12c 00007f52dd3d091b 00007f52dd3d05e3 00007f52dd4cab14 00007f52dd4b6dfa 00007f52dd4badd1 00007f52dd4bad89 00007f52dd4bad19 00007f52dd4bace1
0812 19:13:03.456611 (+ 364us) tablet.cc:1381] finished ApplyRowOperation cycle
0812 19:13:03.456801 (+ 190us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0812 19:13:03.456851 (+ 50us) write_op.cc:312] APPLY: finished
0812 19:13:03.511749 (+ 54898us) log.cc:842] Serialized 23 byte log entry
0812 19:13:03.512051 (+ 302us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:13:03.512200 (+ 149us) write_op.cc:454] Released schema lock
0812 19:13:03.512284 (+ 84us) write_op.cc:341] FINISH: Updating metrics
0812 19:13:03.512651 (+ 367us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:13:03.512678 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":215891,"apply.run_cpu_time_us":6793,"apply.run_wall_time_us":246752,"num_ops":1,"prepare.queue_time_us":372563,"prepare.run_cpu_time_us":6540,"prepare.run_wall_time_us":124418,"replication_time_us":1647824,"spinlock_wait_cycles":3256192,"thread_start_us":124154,"threads_started":1}]]}
I20260812 19:13:03.650439 21233 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:13:03.578369 23111 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:03.713666 21233 raft_consensus.cc:595] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
W20260812 19:13:03.705245 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 137755us
I20260812 19:13:03.710659 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.716810 21244 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:03.706018 21243 raft_consensus.cc:2379] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 2: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:03.717258 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.717435 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.718503 21287 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:03.718436 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.719326 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.720130 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.724695 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.725412 21239 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:03.725672 22983 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:03.726378 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.729919 21033 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48448 (request call id 256) took 4736 ms. Trace:
I20260812 19:13:03.732913 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.733563 23109 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:03.735040 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.736004 23111 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:03.736764 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.652166 23069 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:03.738243 21200 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:03.733747 21152 raft_consensus.cc:2454] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:13:03.724592 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.739523 21200 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:03.733757 21034 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48428 (request call id 246) took 3612 ms. Trace:
I20260812 19:13:03.748214 21112 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:03.741360 21034 rpcz_store.cc:276] 0812 19:13:00.121892 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:00.129955 (+ 8063us) service_pool.cc:227] Handling call
0812 19:13:03.733573 (+3603618us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":129790,"reported_tablets":1}
I20260812 19:13:03.755345 21112 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:03.721123 23024 consensus_meta.cc:287] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.517s user 0.010s sys 0.002s
I20260812 19:13:03.754357 22011 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:03.764106 21200 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:03.667223 23123 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:03.761157 21114 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:03.767138 21200 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:03.761713 21112 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:03.765215 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:03.768332 21287 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:03.768860 21112 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:03.726394 23105 raft_consensus.cc:2726] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:03.770195 21112 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:03.764524 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:03.722942 23132 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:03.749675 23076 raft_consensus.cc:2781] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:03.771886 21200 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:03.772809 22612 raft_consensus.cc:2726] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:03.775976 21112 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:03.779702 22612 raft_consensus.cc:2726] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:03.779758 21204 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:03.781435 22612 raft_consensus.cc:2781] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:03.781844 22612 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:03.782204 22612 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:03.750106 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.786302 21112 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:03.783028 21200 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:03.765013 23058 raft_consensus.cc:2726] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:03.783373 21324 raft_consensus.cc:2454] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:03.761304 22983 raft_consensus.cc:2781] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:03.790256 21200 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:03.792116 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:03.790271 21112 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:03.770371 23117 raft_consensus.cc:2726] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:03.770526 23103 raft_consensus.cc:2726] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:03.775998 23088 raft_consensus.cc:2781] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:03.779433 23120 raft_consensus.cc:2781] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:03.740090 21033 rpcz_store.cc:276] 0812 19:12:58.993329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:58.995528 (+ 2199us) service_pool.cc:227] Handling call
0812 19:13:03.729631 (+4734103us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":4}
I20260812 19:13:03.781527 23062 raft_consensus.cc:2781] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:03.781209 22978 raft_consensus.cc:2781] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260812 19:13:03.822250 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 67977.5us
I20260812 19:13:03.782860 23076 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:03.869474 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 89801.8us
W20260812 19:13:03.861043 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 92772.1us
W20260812 19:13:03.834343 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 75236us
W20260812 19:13:03.858688 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 68482.3us
I20260812 19:13:03.866777 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1539) took 1050 ms. Trace:
I20260812 19:13:03.794196 22997 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:03.792037 23085 raft_consensus.cc:2781] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:03.876617 21287 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:03.877207 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:03.878527 21114 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:03.797840 22983 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:03.882073 21158 raft_consensus.cc:3037] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:03.834028 23077 raft_consensus.cc:2781] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:03.868638 23103 raft_consensus.cc:2726] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:03.867601 22978 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:03.883016 23074 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:03.836143 23120 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:03.886814 23082 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:03.807502 23094 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:03.808562 23069 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:03.913245 22978 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:03.770406 23024 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:03.821784 23088 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:03.895476 21114 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:03.904186 21200 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:03.867834 23090 raft_consensus.cc:2781] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:03.863129 23143 raft_consensus.cc:2781] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:03.872404 23076 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:03.874147 21329 rpcz_store.cc:276] 0812 19:13:02.816622 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:02.816788 (+ 166us) service_pool.cc:227] Handling call
0812 19:13:02.845934 (+ 29146us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:03.691952 (+846018us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:03.866392 (+174440us) spinlock_profiling.cc:243] Waited 817 ms on lock 0x7b540039add4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:03.866647 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1880172800}
I20260812 19:13:03.902634 23077 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:03.917649 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 127428us
I20260812 19:13:03.940290 22053 raft_consensus.cc:1226] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:03.906208 23155 raft_consensus.cc:2781] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:03.868415 23062 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:03.948264 23062 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:03.902215 22983 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:03.949960 23058 raft_consensus.cc:2726] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:03.924520 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.952230 21114 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:03.919795 21287 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:03.890172 23085 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:03.928660 22011 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:03.908469 23117 raft_consensus.cc:2781] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:03.908351 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.918282 23120 raft_consensus.cc:3037] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:03.940706 21112 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:03.943188 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.954085 21114 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:03.956859 23133 log.cc:924] Time spent T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.207s user 0.001s sys 0.000s
W20260812 19:13:03.957531 23140 log.cc:924] Time spent T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260812 19:13:03.686856 23023 rpcz_store.cc:276] 0812 19:12:59.238181 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:59.238451 (+ 270us) service_pool.cc:227] Handling call
0812 19:12:59.575042 (+336591us) ts_tablet_manager.cc:928] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551)
0812 19:13:01.849084 (+2274042us) tablet_copy_client.cc:361] Tablet copy session begun
0812 19:13:03.066321 (+1217237us) tablet_metadata.cc:641] Metadata flushed
0812 19:13:03.066729 (+ 408us) tablet_copy_client.cc:490] Wrote new tablet metadata
0812 19:13:03.527827 (+461098us) tablet_copy_client.cc:411] Wrote new consensus metadata
0812 19:13:03.591666 (+ 63839us) tablet_replica.cc:205] Creating consensus instance
0812 19:13:03.598418 (+ 6752us) ts_tablet_manager.cc:983] Replica registered.
0812 19:13:03.598541 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":5980,"mutex_wait_us":356,"tablet-copy.queue_time_us":173758,"thread_start_us":192222,"threads_started":2}
I20260812 19:13:03.960505 21152 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:13:03.937373 23088 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:03.936434 23090 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:03.986049 22612 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:03.941874 23143 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:03.962721 21287 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:04.002914 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.001840 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:03.944859 23077 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:04.011001 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:04.011396 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:04.011802 23074 raft_consensus.cc:2726] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:04.011879 21237 raft_consensus.cc:2454] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
I20260812 19:13:03.944250 23103 raft_consensus.cc:2781] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:04.015401 22612 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:03.959295 22047 raft_consensus.cc:1226] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:03.999403 23090 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:03.978348 23093 raft_consensus.cc:2726] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:04.025580 23158 raft_consensus.cc:2726] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:04.026456 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:04.025614 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 84986us
I20260812 19:13:03.943883 23155 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:04.030366 21287 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:03.967340 23117 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:03.978771 23137 raft_consensus.cc:2726] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:03.960450 23085 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:04.044405 22051 raft_consensus.cc:2454] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:13:04.015074 23143 raft_consensus.cc:3037] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:04.046085 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.029474 23103 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:04.061671 23103 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:04.052981 23069 raft_consensus.cc:2781] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:04.047739 23155 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:04.052011 21288 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:13:04.042263 23123 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:04.035980 23023 tablet_copy_client.cc:791] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:13:04.066114 21287 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:04.053634 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:04.089874 23137 raft_consensus.cc:2781] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:04.090520 23137 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:04.091076 23137 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:04.092901 21158 raft_consensus.cc:2454] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:04.092896 23049 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:04.072597 21112 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:04.063352 23153 raft_consensus.cc:2726] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:04.037164 23121 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:04.070716 23127 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:04.049309 23117 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:04.096496 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.071247 23024 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:04.105582 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.094275 21287 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:04.078517 23129 log.cc:924] Time spent T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.527s user 0.009s sys 0.010s
I20260812 19:13:04.094904 22896 raft_consensus.cc:2726] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:04.071732 23069 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:04.109107 21287 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:04.109447 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.084122 23074 raft_consensus.cc:2781] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:04.112845 21153 raft_consensus.cc:2379] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:04.108438 23023 tablet_copy_client.cc:655] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:13:04.113970 23116 raft_consensus.cc:2726] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:04.117873 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:04.107910 23160 raft_consensus.cc:2781] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:04.124500 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.115044 21112 leader_election.cc:302] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:04.107124 23082 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:04.130826 21287 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:04.133643 21112 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:04.136608 23049 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:04.138016 23049 raft_consensus.cc:2781] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:04.138578 23049 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:04.135754 23161 raft_consensus.cc:2781] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:04.138980 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.139324 23161 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:04.138060 23153 raft_consensus.cc:2726] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:13:04.117801 23069 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:04.135859 23147 raft_consensus.cc:2781] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:04.141656 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:04.117276 23074 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:04.091128 23132 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:04.148350 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 130577us
I20260812 19:13:04.135262 23158 raft_consensus.cc:2726] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:04.191320 21158 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:04.194250 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:04.194479 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.138995 23049 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:04.195523 21156 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:13:04.194433 23132 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:04.196133 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 65416.9us
I20260812 19:13:04.128808 23160 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:04.181646 23014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c9ea4c9e414d1093d28b6c91e19ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:13:04.200021 22011 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:04.144008 23161 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:04.223055 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:04.239246 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:04.225585 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 86122.7us
I20260812 19:13:04.221643 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:04.227783 21287 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:04.185328 23074 raft_consensus.cc:3037] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:04.229485 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:04.244567 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:04.246338 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:04.248445 21156 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:13:04.195648 23082 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:04.250528 21324 raft_consensus.cc:2379] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 2: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:04.251828 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:04.200471 23147 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:04.250643 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:04.227659 23160 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:04.249173 21156 raft_consensus.cc:595] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:13:04.261449 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.240155 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:04.263240 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:04.265142 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 361080us
I20260812 19:13:04.282945 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.246403 23132 raft_consensus.cc:2781] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:04.284626 21200 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:04.248471 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:04.261759 23147 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
W20260812 19:13:04.291569 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 337565us
W20260812 19:13:04.290257 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 78419.3us
I20260812 19:13:04.293241 21114 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
W20260812 19:13:04.290666 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 62362.9us
I20260812 19:13:04.294312 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.297137 22011 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:04.299990 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.301754 21328 raft_consensus.cc:1226] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:04.244247 22983 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:04.307621 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:04.299371 23104 consensus_meta.cc:287] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.764s user 0.005s sys 0.009s
I20260812 19:13:04.288251 23132 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:04.307755 21200 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:04.261977 23082 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:04.313324 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.317446 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.317620 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.318300 21149 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 4.
I20260812 19:13:04.318849 21158 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:04.323220 21243 raft_consensus.cc:1226] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:04.315985 23104 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:04.327553 21150 raft_consensus.cc:1226] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 1. Current term is 2. Ops: []
W20260812 19:13:04.317847 23075 consensus_meta.cc:287] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.641s user 0.013s sys 0.000s
W20260812 19:13:04.336000 23159 log.cc:924] Time spent T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
I20260812 19:13:04.336194 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.285933 23155 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:04.314203 23024 raft_consensus.cc:2781] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:04.342941 23024 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:04.315621 23132 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:04.345491 23024 raft_consensus.cc:3037] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:04.346786 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.348031 21151 raft_consensus.cc:1226] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:04.320673 23127 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:04.344296 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:04.303434 23120 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:04.320442 23153 raft_consensus.cc:2781] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:04.355173 22046 raft_consensus.cc:1226] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
W20260812 19:13:04.355809 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 64298.4us
I20260812 19:13:04.343302 23155 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:04.361280 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:04.380769 21155 raft_consensus.cc:1226] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
W20260812 19:13:04.362987 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 69818.8us
I20260812 19:13:04.329912 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:04.381672 21114 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:04.383200 21114 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:04.383401 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:04.384440 21321 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:04.385712 21112 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:04.354198 21323 raft_consensus.cc:1226] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:04.391887 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:04.357980 23153 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:04.395452 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:04.397452 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.397861 23171 raft_consensus.cc:2781] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:04.398463 23171 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:04.398934 23171 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:04.345058 23075 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:04.403740 21115 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:04.407761 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 100063us
I20260812 19:13:04.399631 23153 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:04.415114 21200 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:04.417205 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:04.417335 21200 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:04.364795 22978 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:04.418648 22052 raft_consensus.cc:1226] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:04.418745 23174 raft_consensus.cc:2726] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:04.419409 23169 consensus_queue.cc:1046] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.110s
I20260812 19:13:04.419669 23062 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:04.421766 23073 consensus_queue.cc:1046] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260812 19:13:04.430855 23116 raft_consensus.cc:2726] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:04.431891 21237 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49361681c5c1421594d10cd52061ec44"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:13:04.437958 22983 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:04.438123 23170 raft_consensus.cc:2781] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:04.436730 21200 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:04.443193 23173 raft_consensus.cc:2781] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:04.399650 23143 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:04.438414 23132 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:04.421074 23088 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:04.445278 23075 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:04.439038 23082 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:04.428473 23058 raft_consensus.cc:2781] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:04.448416 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:04.433782 23147 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:04.410597 23175 raft_consensus.cc:2781] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:04.462200 21237 raft_consensus.cc:595] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:13:04.464869 23104 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:04.444972 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 147913us
I20260812 19:13:04.509153 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:04.494748 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 58161.1us
I20260812 19:13:04.421177 23076 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:04.454100 22978 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:04.504112 21329 raft_consensus.cc:1226] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:04.462144 23172 raft_consensus.cc:2726] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:13:04.520553 21200 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:04.490384 23170 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:04.528434 21200 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:04.530476 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:04.525745 23109 raft_consensus.cc:2781] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:04.537536 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:04.538645 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 153013us
I20260812 19:13:04.526741 22048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40614 (request call id 341) took 2188 ms. Trace:
I20260812 19:13:04.535130 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:04.535670 23062 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:04.502012 23058 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:04.537791 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:04.538749 21322 raft_consensus.cc:2379] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 2: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:04.534505 23170 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:04.513825 21237 raft_consensus.cc:603] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:13:04.551546 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:04.556537 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:04.489770 23173 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:04.559613 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:04.560582 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 50508.6us
I20260812 19:13:04.519273 23175 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:04.578258 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:13:04.610165 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 47049.3us
I20260812 19:13:04.629179 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:04.625041 22008 leader_election.cc:334] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.540s (SENT)
I20260812 19:13:04.563961 21327 raft_consensus.cc:1226] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
W20260812 19:13:04.594887 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 211712us
I20260812 19:13:04.573711 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:04.581827 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:04.645536 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:04.575285 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:04.645812 22008 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:04.710083 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:04.578807 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:05.081980 21155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 156) took 1077 ms. Trace:
I20260812 19:13:04.557436 23058 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:04.556352 23120 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:04.566083 23173 raft_consensus.cc:3037] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:04.781973 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 469291us
I20260812 19:13:04.785655 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:04.649513 23147 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:04.645887 23175 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:04.648418 22056 raft_consensus.cc:1226] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:04.647439 21114 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:05.060457 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 409343us
I20260812 19:13:04.647549 21328 raft_consensus.cc:2379] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:04.557189 22048 rpcz_store.cc:276] 0812 19:13:02.338135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:02.342285 (+ 4150us) service_pool.cc:227] Handling call
0812 19:13:03.090699 (+748414us) spinlock_profiling.cc:243] Waited 8.2 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52c6c 00007f52d6b52bd6 000000000034aea9 00007f52de972a30 00007f52dea387b4 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:03.090938 (+ 239us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:03.091271 (+ 333us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:13:03.230484 (+139213us) log.cc:842] Serialized 14 byte log entry
0812 19:13:03.727891 (+497407us) spinlock_profiling.cc:243] Waited 7.67 ms on lock 0x7b18002f2960. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d973582c 00007f52d6b6c9dc 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b89e22 00007f52d97275bb 00007f52d972d063 00007f52d972d57b
0812 19:13:04.048505 (+320614us) spinlock_profiling.cc:243] Waited 5.4 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:04.292034 (+243529us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b18002f2960. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b6c345 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b87155 00007f52d9a9e3d0 00007f52d9ab0576 00007f52d9ac10d7 00007f52d9ac1058
0812 19:13:04.514383 (+222349us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:04.521974 (+ 7591us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:04.525253 (+ 3279us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:04.526406 (+ 1153us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:04.526593 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12570,"prepare.queue_time_us":365897,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":57,"raft.queue_time_us":25151,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":8810,"spinlock_wait_cycles":73850240,"thread_start_us":613974,"threads_started":3,"wal-append.queue_time_us":313858}
I20260812 19:13:05.061007 23181 consensus_queue.cc:1046] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:13:04.598151 21151 raft_consensus.cc:1226] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
W20260812 19:13:05.066324 23160 consensus_meta.cc:287] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.775s user 0.005s sys 0.014s
I20260812 19:13:05.064683 21320 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:05.067000 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 538603us
I20260812 19:13:05.069090 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:05.137343 21200 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:05.070160 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:05.071666 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:05.071503 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1325) took 1313 ms. Trace:
I20260812 19:13:04.803603 23148 raft_consensus.cc:2726] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:04.629065 23169 raft_consensus.cc:2781] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260812 19:13:05.078297 23077 consensus_meta.cc:287] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.043s user 0.012s sys 0.008s
W20260812 19:13:04.998768 21115 consensus_peers.cc:487] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c31c6a2accba4d29ba096500b9d1a911. This is attempt 1: this message will repeat every 5th retry.
W20260812 19:13:05.132354 23161 consensus_meta.cc:287] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.881s user 0.001s sys 0.018s
I20260812 19:13:05.082422 21155 rpcz_store.cc:276] 0812 19:13:04.004879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:04.005008 (+ 129us) service_pool.cc:227] Handling call
0812 19:13:04.005294 (+ 286us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:04.386654 (+381360us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:04.830268 (+443614us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b5400181854. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:05.081916 (+251648us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":863053568}
I20260812 19:13:05.066341 21150 raft_consensus.cc:1226] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 3. Ops: []
W20260812 19:13:05.107975 23024 consensus_meta.cc:287] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.754s user 0.010s sys 0.007s
I20260812 19:13:05.138482 21243 rpcz_store.cc:276] 0812 19:13:03.758247 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:03.759204 (+ 957us) service_pool.cc:227] Handling call
0812 19:13:03.772588 (+ 13384us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:04.325939 (+553351us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:04.818821 (+492882us) spinlock_profiling.cc:243] Waited 550 ms on lock 0x7b54000b6154. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:05.071373 (+252552us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1265819392}
W20260812 19:13:05.133389 23069 consensus_meta.cc:287] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.953s user 0.010s sys 0.010s
W20260812 19:13:05.133628 23090 consensus_meta.cc:287] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.086s user 0.008s sys 0.012s
W20260812 19:13:05.128664 23153 consensus_meta.cc:287] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.705s user 0.008s sys 0.007s
W20260812 19:13:05.161796 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 537357us
I20260812 19:13:05.220494 22612 consensus_queue.cc:1046] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:13:05.094978 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:05.149080 22052 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 519) took 1484 ms. Trace:
I20260812 19:13:05.183614 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:05.216192 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:13:05.147902 23137 consensus_meta.cc:287] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 1.056s user 0.011s sys 0.010s
I20260812 19:13:05.183786 23077 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:05.210783 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 563388us
I20260812 19:13:05.224984 21155 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:05.172569 23094 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:05.161491 23169 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:05.166280 23132 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:05.184180 23082 raft_consensus.cc:2726] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:05.226783 23023 tablet_copy_client.cc:531] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:13:05.167008 23143 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:05.167168 23088 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:05.225556 23153 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:05.229388 23173 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.235525 23076 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:05.299561 21287 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:05.235078 22047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 520) took 1583 ms. Trace:
I20260812 19:13:05.298421 22048 raft_consensus.cc:1226] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:05.211326 23161 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:05.315608 21156 raft_consensus.cc:2379] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:13:05.317257 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 697438us
I20260812 19:13:05.221565 23090 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:05.282928 23169 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
W20260812 19:13:05.299813 23049 consensus_meta.cc:287] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 1.073s user 0.011s sys 0.006s
I20260812 19:13:05.150200 23160 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:05.279865 22046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 523) took 1596 ms. Trace:
I20260812 19:13:05.301630 23073 raft_consensus.cc:3032] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:05.341711 21287 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:05.228905 23024 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:05.269853 23164 consensus_queue.cc:1046] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260812 19:13:05.299654 23188 raft_consensus.cc:2781] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:05.347968 21233 raft_consensus.cc:1226] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:05.234920 23069 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.316684 23153 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:05.354910 21287 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:05.254110 23137 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:05.299474 23183 raft_consensus.cc:2781] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:05.357544 22008 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:05.298945 23171 consensus_meta.cc:287] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.899s user 0.006s sys 0.009s
I20260812 19:13:05.309621 23163 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:05.226238 22052 rpcz_store.cc:276] 0812 19:13:03.648474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:03.649409 (+ 935us) service_pool.cc:227] Handling call
0812 19:13:03.649835 (+ 426us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:04.437204 (+787369us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:04.821855 (+384651us) spinlock_profiling.cc:243] Waited 761 ms on lock 0x7b54006e0fd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:05.132818 (+310963us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1751165312}
I20260812 19:13:05.359862 23156 raft_consensus.cc:2726] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 19:13:05.421420 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 259312us
I20260812 19:13:05.393489 23176 raft_consensus.cc:3032] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:05.353516 23073 raft_consensus.cc:729] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:13:05.419474 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 62065.9us
I20260812 19:13:05.299316 23136 consensus_queue.cc:1046] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.252s
I20260812 19:13:05.407138 22983 raft_consensus.cc:3032] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:05.431318 21324 raft_consensus.cc:1226] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:05.431622 21319 raft_consensus.cc:1226] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:05.472097 22008 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:05.439417 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 307865us
I20260812 19:13:05.353482 23049 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:05.408981 23188 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:05.362196 23094 consensus_queue.cc:1046] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.115s
I20260812 19:13:05.489892 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50042 (request call id 186) took 1214 ms. Trace:
I20260812 19:13:05.489941 22008 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:05.490641 21324 rpcz_store.cc:276] 0812 19:13:04.275527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:04.286583 (+ 11056us) service_pool.cc:227] Handling call
0812 19:13:04.286963 (+ 380us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:05.478348 (+1191385us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:05.489524 (+ 11176us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54002314d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:05.489803 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2631097728}
I20260812 19:13:05.491555 23132 raft_consensus.cc:2726] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:05.492283 23132 raft_consensus.cc:2726] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:05.493057 23094 raft_consensus.cc:3032] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:05.493594 23094 raft_consensus.cc:729] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:05.493695 23082 consensus_queue.cc:1046] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:13:05.494918 23082 raft_consensus.cc:3032] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:05.495314 23082 raft_consensus.cc:729] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:05.494769 23094 consensus_queue.cc:260] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.496029 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:05.496232 23082 consensus_queue.cc:260] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.492137 22008 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:05.470047 23074 consensus_meta.cc:287] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 1.213s user 0.021s sys 0.000s
I20260812 19:13:05.492960 23132 consensus_queue.cc:1046] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:13:05.332347 22047 rpcz_store.cc:276] 0812 19:13:03.651740 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:03.656133 (+ 4393us) service_pool.cc:227] Handling call
0812 19:13:03.724284 (+ 68151us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:04.044812 (+320528us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:05.167504 (+1122692us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b540058e654. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:05.227113 (+ 59609us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":488968448}
I20260812 19:13:05.511755 21151 raft_consensus.cc:1226] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:05.422482 23179 raft_consensus.cc:3032] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:05.514851 22011 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
W20260812 19:13:05.516103 23151 consensus_queue.cc:1159] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f741102bfe1441d1b453e5a838649059" responder_term: 2 status { last_received { term: 2 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 2 index: 5 } }
I20260812 19:13:05.517060 23088 raft_consensus.cc:3032] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:05.517478 23088 raft_consensus.cc:729] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:05.518390 23088 consensus_queue.cc:260] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.518551 22612 consensus_queue.cc:1046] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:13:05.519994 23088 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:05.520059 22612 raft_consensus.cc:3032] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:05.520612 22612 raft_consensus.cc:729] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:05.521543 22612 consensus_queue.cc:260] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.524293 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:05.525030 21236 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:05.525074 21156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1514) took 1290 ms. Trace:
I20260812 19:13:05.525493 21156 rpcz_store.cc:276] 0812 19:13:04.209168 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:04.305643 (+ 96475us) service_pool.cc:227] Handling call
0812 19:13:05.499367 (+1193724us) spinlock_profiling.cc:243] Waited 997 ms on lock 0x7b5400325354. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:05.499595 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2293241088}
I20260812 19:13:05.430380 23176 raft_consensus.cc:729] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:05.432967 23111 raft_consensus.cc:2726] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:05.441123 23109 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:05.530879 21318 raft_consensus.cc:1226] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:05.431190 23183 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:05.535080 22008 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:05.540328 22612 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:05.417956 23171 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:05.544356 23171 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:05.477919 22983 raft_consensus.cc:729] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:05.409410 23058 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.537654 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:05.524379 23085 consensus_meta.cc:287] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.456s user 0.020s sys 0.001s
W20260812 19:13:05.527446 23017 consensus_meta.cc:287] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.228s user 0.014s sys 0.005s
I20260812 19:13:05.571334 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1302) took 1189 ms. Trace:
W20260812 19:13:05.536024 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 52302us
I20260812 19:13:05.486011 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:05.500913 23188 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
W20260812 19:13:05.546252 23103 consensus_meta.cc:287] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.466s user 0.014s sys 0.006s
I20260812 19:13:05.524770 23179 raft_consensus.cc:729] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:05.417784 23163 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:05.554925 23117 consensus_meta.cc:287] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.447s user 0.014s sys 0.007s
I20260812 19:13:05.545408 23082 raft_consensus.cc:3037] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:05.540436 23077 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:05.589412 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1308) took 1041 ms. Trace:
I20260812 19:13:05.590036 21318 rpcz_store.cc:276] 0812 19:13:04.548486 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:04.549201 (+ 715us) service_pool.cc:227] Handling call
0812 19:13:04.571113 (+ 21912us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:05.538300 (+967187us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:05.589101 (+ 50801us) spinlock_profiling.cc:243] Waited 959 ms on lock 0x7b5400230fd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:05.589342 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2206656384}
I20260812 19:13:05.547155 23183 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:05.350236 22046 rpcz_store.cc:276] 0812 19:13:03.683124 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:03.685107 (+ 1983us) service_pool.cc:227] Handling call
0812 19:13:03.727561 (+ 42454us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:04.393264 (+665703us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:05.234923 (+841659us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b5400452254. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:05.269326 (+ 34403us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1427811072}
I20260812 19:13:05.547545 23109 raft_consensus.cc:3037] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:05.584815 21115 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:05.584530 21235 raft_consensus.cc:1226] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:05.598640 21236 raft_consensus.cc:2454] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
W20260812 19:13:05.589294 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 351167us
I20260812 19:13:05.613580 21114 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:05.605822 23197 raft_consensus.cc:2726] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:13:05.606848 23195 raft_consensus.cc:2726] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:05.609071 23049 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:05.580168 23132 raft_consensus.cc:2726] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:05.609818 23073 consensus_queue.cc:260] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.618340 21156 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:13:05.527240 23074 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.571939 21319 rpcz_store.cc:276] 0812 19:13:04.381892 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:04.382056 (+ 164us) service_pool.cc:227] Handling call
0812 19:13:04.397528 (+ 15472us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:05.478508 (+1080980us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:05.570955 (+ 92447us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400355054. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:05.571251 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2377712512}
I20260812 19:13:05.496268 23094 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:05.622222 21203 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:05.624269 21114 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:05.623764 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 456544us
I20260812 19:13:05.632731 23158 raft_consensus.cc:2726] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:05.633306 23167 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:05.633237 23116 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:05.628515 23190 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:05.641160 23167 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.641467 23190 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:05.641103 21156 raft_consensus.cc:595] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Received request to transfer leadership to TS d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:05.643599 21156 raft_consensus.cc:603] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:13:05.569295 23085 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.618364 22978 raft_consensus.cc:2726] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:05.625011 23173 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:05.658160 21200 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:05.590139 23103 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:05.646278 23170 consensus_meta.cc:287] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.074s user 0.016s sys 0.001s
I20260812 19:13:05.593200 23117 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:05.635968 23090 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:05.644493 23161 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:05.691146 21157 raft_consensus.cc:1226] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:05.651625 23195 raft_consensus.cc:2726] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:13:05.721415 21152 raft_consensus.cc:2379] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:05.650467 23024 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:05.649874 23158 raft_consensus.cc:2726] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:05.650719 23160 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:05.641172 23116 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.707634 23069 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:05.731801 23196 raft_consensus.cc:2781] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:05.617206 23176 consensus_queue.cc:260] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.754993 21154 raft_consensus.cc:2379] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:13:05.741415 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 226705us
I20260812 19:13:05.633714 23120 consensus_queue.cc:1046] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.084s
I20260812 19:13:05.683177 23171 raft_consensus.cc:2781] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260812 19:13:05.748553 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 393735us
I20260812 19:13:05.628922 22983 consensus_queue.cc:260] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:05.751042 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 216105us
W20260812 19:13:05.713187 21320 consensus_meta.cc:287] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.576s user 0.006s sys 0.011s
I20260812 19:13:05.761988 21320 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
W20260812 19:13:05.746694 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 59897.6us
I20260812 19:13:05.756488 22011 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
W20260812 19:13:05.757253 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 99250.8us
I20260812 19:13:05.655270 23179 consensus_queue.cc:260] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.759038 23069 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:05.724279 22896 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:05.813606 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 426129us
I20260812 19:13:05.733717 23017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:40614 (request call id 319) took 6832 ms. Trace:
I20260812 19:13:05.734133 23137 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:05.816087 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:05.830824 21237 raft_consensus.cc:3037] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:05.820564 21155 consensus_meta.cc:287] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.549s user 0.010s sys 0.001s
W20260812 19:13:05.821388 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 60434us
I20260812 19:13:05.821712 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1310) took 1213 ms. Trace:
W20260812 19:13:05.822301 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 58696.4us
I20260812 19:13:05.734894 23024 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:05.750346 23196 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:05.741469 23190 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:05.828486 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 66772us
I20260812 19:13:05.834946 21320 rpcz_store.cc:276] 0812 19:13:04.549076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:04.558192 (+ 9116us) service_pool.cc:227] Handling call
0812 19:13:05.762683 (+1204491us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:05.694705 23170 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.751498 23184 raft_consensus.cc:3032] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:05.851284 21240 raft_consensus.cc:1226] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:05.834757 21287 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:05.842139 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:05.836370 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:05.839265 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 82874.7us
I20260812 19:13:05.844652 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:05.852522 21329 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:05.852872 21321 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:05.853878 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:05.835150 21155 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:13:05.847745 22008 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:05.848857 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:05.858371 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 179042us
I20260812 19:13:05.860335 21203 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:05.864140 21320 raft_consensus.cc:2379] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:05.864461 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:05.814150 23143 raft_consensus.cc:2726] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:05.743084 23153 consensus_queue.cc:1046] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.095s
I20260812 19:13:05.814177 23171 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:05.815333 23161 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:05.897450 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:05.898360 21238 raft_consensus.cc:1226] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:05.832707 23058 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:05.911731 21150 raft_consensus.cc:1226] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:05.911722 21233 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:05.897363 21239 raft_consensus.cc:1226] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:05.851209 23184 raft_consensus.cc:729] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:05.871235 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:05.867256 21287 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:05.865628 23062 raft_consensus.cc:2726] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:05.871395 23190 raft_consensus.cc:2726] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:05.867281 23024 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.830235 22896 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.954231 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:05.899870 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 52217.4us
I20260812 19:13:05.852399 23196 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:05.837249 23167 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:05.849133 23164 raft_consensus.cc:2726] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:13:05.939852 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:05.912544 23171 raft_consensus.cc:3037] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:05.823444 23069 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.691112 23194 consensus_queue.cc:1046] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.222s
I20260812 19:13:05.922938 23176 raft_consensus.cc:3037] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:05.942787 23136 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:05.946758 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:05.947295 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 80148.8us
W20260812 19:13:05.949549 21152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1608) took 1904 ms (client timeout 1714 ms). Trace:
I20260812 19:13:05.928568 23205 raft_consensus.cc:3037] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:05.923883 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:05.987277 21322 raft_consensus.cc:2379] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:05.914336 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:05.958431 21238 raft_consensus.cc:1226] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 2. Current term is 3. Ops: []
W20260812 19:13:05.960140 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 94491.4us
I20260812 19:13:05.988600 21203 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:05.965374 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:05.966243 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:05.970206 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:05.971062 21155 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:13:05.971287 21239 raft_consensus.cc:1226] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:05.956835 23173 raft_consensus.cc:2781] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:05.990111 21203 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
W20260812 19:13:05.899011 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 46157.6us
I20260812 19:13:05.990900 22056 raft_consensus.cc:3037] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:05.945628 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:05.991180 21155 raft_consensus.cc:595] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Received request to transfer leadership to TS d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:05.957805 23156 raft_consensus.cc:2781] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:05.982824 21287 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
W20260812 19:13:05.982477 21152 rpcz_store.cc:269] 0812 19:13:04.045625 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:04.045785 (+ 160us) service_pool.cc:227] Handling call
0812 19:13:05.949200 (+1903415us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b54003cf854. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:05.949444 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3799177600}
I20260812 19:13:05.923528 23161 raft_consensus.cc:504] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:05.940512 22983 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:05.991968 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:05.990998 21328 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:05.972148 23163 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:05.961232 23213 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:05.963302 23074 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:05.995046 21203 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:05.998030 22011 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:05.998522 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:05.957528 23177 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:05.974471 23085 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:05.971441 23127 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:06.025015 21287 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:05.999989 23194 raft_consensus.cc:2726] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:06.026266 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:06.036460 21244 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:06.033792 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260812 19:13:06.039881 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 44931.1us
I20260812 19:13:05.990377 23103 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:05.994331 23117 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:06.040557 22048 raft_consensus.cc:2379] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 2: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:06.023384 23156 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:06.044770 21238 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:06.044426 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 46493.3us
I20260812 19:13:06.045238 21203 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:05.825150 23017 rpcz_store.cc:276] 0812 19:12:58.899097 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:58.899424 (+ 327us) service_pool.cc:227] Handling call
0812 19:12:59.300618 (+401194us) ts_tablet_manager.cc:928] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
0812 19:13:03.480765 (+4180147us) tablet_copy_client.cc:361] Tablet copy session begun
0812 19:13:04.042740 (+561975us) tablet_metadata.cc:641] Metadata flushed
0812 19:13:04.061866 (+ 19126us) tablet_copy_client.cc:490] Wrote new tablet metadata
0812 19:13:05.577834 (+1515968us) tablet_copy_client.cc:411] Wrote new consensus metadata
0812 19:13:05.636773 (+ 58939us) tablet_replica.cc:205] Creating consensus instance
0812 19:13:05.684730 (+ 47957us) ts_tablet_manager.cc:983] Replica registered.
0812 19:13:05.731453 (+ 46723us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":464,"tablet-copy.queue_time_us":236491,"thread_start_us":475250,"threads_started":2}
I20260812 19:13:05.995616 23173 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:06.046491 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:13:06.071201 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:06.086156 21240 raft_consensus.cc:2379] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 2: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:06.010525 23212 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:06.048130 23074 raft_consensus.cc:2726] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:06.081895 21203 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:05.997443 21323 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:06.034638 23049 consensus_queue.cc:1046] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260812 19:13:06.047676 23116 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:06.104895 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 46009.8us
I20260812 19:13:06.096135 22011 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:06.011235 23222 raft_consensus.cc:2781] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:13:06.109052 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:13:06.110433 21287 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:06.103875 23017 tablet_copy_client.cc:791] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:13:06.110584 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:06.056823 23151 raft_consensus.cc:2781] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:13:06.047255 23208 raft_consensus.cc:2726] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:06.083448 23156 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:06.052994 23127 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:05.994275 23136 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:06.044373 23153 consensus_queue.cc:1046] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260812 19:13:06.107820 23173 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:06.104504 23093 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:06.039744 23213 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:06.147976 23213 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:06.140170 22008 leader_election.cc:334] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.833s (SENT)
I20260812 19:13:06.108704 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:06.057485 23085 consensus_queue.cc:1046] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260812 19:13:06.137398 23137 raft_consensus.cc:2726] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:06.047050 23198 consensus_queue.cc:1046] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.097s
I20260812 19:13:06.152488 23164 raft_consensus.cc:2781] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election won for term 2
W20260812 19:13:06.144419 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 48424.3us
I20260812 19:13:06.156755 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:06.116317 23212 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:06.142057 23017 tablet_copy_client.cc:655] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:13:06.110229 23143 consensus_queue.cc:1046] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260812 19:13:06.141548 23151 raft_consensus.cc:686] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:06.155223 23224 raft_consensus.cc:2726] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:13:06.159837 23222 raft_consensus.cc:686] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:06.155771 22896 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:06.163849 23024 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:06.222164 23208 raft_consensus.cc:2781] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:06.211431 23069 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:06.261735 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:06.144950 23116 consensus_queue.cc:1046] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.085s
W20260812 19:13:06.244760 23088 consensus_meta.cc:287] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.724s user 0.012s sys 0.003s
I20260812 19:13:06.277405 23225 raft_consensus.cc:3032] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:06.284260 23116 raft_consensus.cc:3032] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:06.284823 23116 raft_consensus.cc:729] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:06.252079 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:06.285846 23116 consensus_queue.cc:260] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:06.143559 23163 consensus_queue.cc:1046] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20260812 19:13:06.223259 23202 raft_consensus.cc:2726] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:06.045154 23177 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:06.302276 21235 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 142) took 1228 ms. Trace:
I20260812 19:13:06.302790 21235 rpcz_store.cc:276] 0812 19:13:05.074133 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:05.074907 (+ 774us) service_pool.cc:227] Handling call
0812 19:13:05.075344 (+ 437us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:05.612850 (+537506us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:06.301979 (+689129us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b5400000d54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:06.302210 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1170553216}
I20260812 19:13:06.156384 23093 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:06.247031 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:06.265192 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:06.299484 21115 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:06.307632 21237 raft_consensus.cc:2454] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:06.158578 23194 consensus_queue.cc:1046] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.084s
I20260812 19:13:06.309437 21233 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:06.283509 23225 raft_consensus.cc:729] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:06.264336 23208 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:06.316551 23136 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:06.291066 23116 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:06.317294 21157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 160) took 2117 ms. Trace:
I20260812 19:13:06.241240 23204 consensus_queue.cc:1046] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.285s
I20260812 19:13:06.325610 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:06.326321 23197 raft_consensus.cc:3032] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:06.326663 22050 raft_consensus.cc:2379] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:06.326747 23197 raft_consensus.cc:729] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:06.323504 21157 rpcz_store.cc:276] 0812 19:13:04.191365 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:04.191843 (+ 478us) service_pool.cc:227] Handling call
0812 19:13:04.192171 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:05.714377 (+1522206us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:06.308943 (+594566us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54003cf354. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:06.309158 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3437899008}
I20260812 19:13:06.292116 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:06.295958 23161 leader_election.cc:288] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:06.311561 23203 consensus_queue.cc:1046] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260812 19:13:06.313935 23085 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:06.344750 21238 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:06.344970 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:06.339493 23212 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:06.329594 23208 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:06.320694 23151 consensus_queue.cc:237] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:06.338052 23220 raft_consensus.cc:3032] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:06.399039 23220 raft_consensus.cc:729] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:13:06.359900 23175 consensus_meta.cc:287] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.222s user 0.011s sys 0.011s
I20260812 19:13:06.406193 21238 raft_consensus.cc:1226] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 3. Ops: []
W20260812 19:13:06.350778 23094 consensus_meta.cc:287] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.704s user 0.009s sys 0.006s
I20260812 19:13:06.358135 23163 raft_consensus.cc:3032] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:06.413086 21243 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:06.413867 23127 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:06.328621 23199 consensus_queue.cc:1046] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.299s
I20260812 19:13:06.419957 23093 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:06.421340 23170 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:06.422646 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:06.327973 23197 consensus_queue.cc:260] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:06.340189 23204 consensus_queue.cc:1046] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
I20260812 19:13:06.409250 23206 ts_tablet_manager.cc:927] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551)
I20260812 19:13:06.366734 22046 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
W20260812 19:13:06.433110 23188 consensus_meta.cc:287] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.844s user 0.009s sys 0.011s
I20260812 19:13:06.433296 23173 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:06.401490 23169 consensus_meta.cc:287] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.986s user 0.012s sys 0.009s
I20260812 19:13:06.440384 22056 raft_consensus.cc:2454] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:06.445226 23023 tablet_bootstrap.cc:492] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:13:06.436925 21157 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:06.462967 23200 raft_consensus.cc:3032] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:06.463487 23200 raft_consensus.cc:729] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:06.449668 21328 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:06.460497 22053 raft_consensus.cc:1226] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
W20260812 19:13:06.449620 23109 consensus_meta.cc:287] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.821s user 0.007s sys 0.008s
I20260812 19:13:06.414525 23163 raft_consensus.cc:729] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:06.403167 23220 consensus_queue.cc:260] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:06.479480 23197 raft_consensus.cc:3037] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:06.447932 23203 raft_consensus.cc:3032] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:06.485790 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
W20260812 19:13:06.488287 21112 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.031s sys 0.080s
W20260812 19:13:06.471714 21321 consensus_meta.cc:287] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.573s user 0.008s sys 0.008s
I20260812 19:13:06.489687 21321 raft_consensus.cc:2454] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
W20260812 19:13:06.489189 21112 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.554s user 0.031s sys 0.080s
I20260812 19:13:06.414861 23175 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:06.500685 21154 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1609) took 2412 ms (client timeout 1811 ms). Trace:
W20260812 19:13:06.501149 21154 rpcz_store.cc:269] 0812 19:13:04.074525 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:04.074768 (+ 243us) service_pool.cc:227] Handling call
0812 19:13:06.486944 (+2412176us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54001955d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:06.487180 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3746888064}
W20260812 19:13:06.512493 23196 consensus_meta.cc:287] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.539s user 0.015s sys 0.000s
I20260812 19:13:06.526259 21112 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:06.487480 23203 raft_consensus.cc:729] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:06.464660 23200 consensus_queue.cc:260] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:06.476194 23109 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:06.483556 23171 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:06.538398 23185 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:06.538928 23170 raft_consensus.cc:2726] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:06.539558 21112 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:06.544260 23177 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:06.537109 23183 consensus_meta.cc:287] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.895s user 0.009s sys 0.010s
I20260812 19:13:06.547545 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57682 (request call id 148) took 1052 ms. Trace:
I20260812 19:13:06.560060 21243 rpcz_store.cc:276] 0812 19:13:05.494885 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:05.497482 (+ 2597us) service_pool.cc:227] Handling call
0812 19:13:06.547313 (+1049831us) spinlock_profiling.cc:243] Waited 740 ms on lock 0x7b5400303c54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:06.547464 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1702911360}
W20260812 19:13:06.560801 21329 consensus_meta.cc:287] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.694s user 0.019s sys 0.007s
I20260812 19:13:06.468025 23169 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:06.544947 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:06.554738 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:06.556330 21328 raft_consensus.cc:1226] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:06.548436 23203 consensus_queue.cc:260] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:06.561143 21329 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:06.466789 23188 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:06.561757 23082 consensus_meta.cc:287] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.967s user 0.012s sys 0.003s
I20260812 19:13:06.415019 23222 consensus_queue.cc:237] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:06.481240 23225 consensus_queue.cc:260] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:06.562812 21324 raft_consensus.cc:1226] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
W20260812 19:13:06.542762 23014 consensus_meta.cc:287] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.913s user 0.016s sys 0.002s
I20260812 19:13:06.564393 21154 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
W20260812 19:13:07.079859 21204 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.002s user 0.018s sys 0.064s
I20260812 19:13:06.595839 21233 raft_consensus.cc:1226] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 1. Current term is 2. Ops: []
W20260812 19:13:07.195686 21287 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.058s user 0.002s sys 0.024s
W20260812 19:13:07.091466 21200 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.000s user 0.018s sys 0.031s
W20260812 19:13:06.652563 21244 consensus_meta.cc:287] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.606s user 0.009s sys 0.012s
I20260812 19:13:06.683542 21236 raft_consensus.cc:1226] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
W20260812 19:13:06.908740 21318 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.980s user 0.061s sys 0.090s
W20260812 19:13:07.197965 21288 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.002s user 0.006s sys 0.007s
W20260812 19:13:07.208395 21288 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.016s user 0.007s sys 0.007s
W20260812 19:13:07.206593 21287 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.069s user 0.002s sys 0.025s
W20260812 19:13:07.045884 22008 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.888s user 0.010s sys 0.047s
W20260812 19:13:07.216287 22008 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.059s user 0.013s sys 0.050s
W20260812 19:13:07.217308 21318 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.279s user 0.061s sys 0.108s
W20260812 19:13:07.206230 21204 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.128s user 0.022s sys 0.066s
W20260812 19:13:07.217425 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 1.05999e+06us
I20260812 19:13:07.195686 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:07.216252 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:07.207216 21200 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.116s user 0.024s sys 0.034s
I20260812 19:13:07.219287 21321 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:07.216648 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 1.07961e+06us
W20260812 19:13:07.219580 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 1.12871e+06us
I20260812 19:13:07.220963 21287 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:07.221441 22011 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:07.207001 21329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1579) took 1365 ms (client timeout 1580 ms). Trace:
I20260812 19:13:07.210251 21157 raft_consensus.cc:1226] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
W20260812 19:13:07.218573 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 1.14086e+06us
W20260812 19:13:07.072073 21203 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.960s user 0.018s sys 0.030s
I20260812 19:13:06.671129 23024 raft_consensus.cc:979] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa: : Instructing follower 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:13:07.223156 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:07.206676 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1340) took 1250 ms. Trace:
I20260812 19:13:06.540936 23196 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:06.549446 23181 consensus_queue.cc:1046] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.759s
I20260812 19:13:07.206959 21244 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
W20260812 19:13:07.445128 21244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1341) took 1478 ms (client timeout 1634 ms). Trace:
I20260812 19:13:07.446197 21154 raft_consensus.cc:3037] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:07.110069 23197 consensus_meta.cc:287] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.630s user 0.013s sys 0.001s
W20260812 19:13:07.223909 21329 rpcz_store.cc:269] 0812 19:13:05.841580 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:05.841747 (+ 167us) service_pool.cc:227] Handling call
0812 19:13:07.206939 (+1365192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:07.449051 21320 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:07.297380 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:07.297991 21321 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
W20260812 19:13:07.446867 21203 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.339s user 0.037s sys 0.056s
I20260812 19:13:07.379487 21322 raft_consensus.cc:1226] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
W20260812 19:13:07.198843 23116 consensus_meta.cc:287] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.865s user 0.012s sys 0.003s
W20260812 19:13:07.193127 23151 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.812s user 0.007s sys 0.005s
W20260812 19:13:07.014111 23205 consensus_meta.cc:287] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.027s user 0.014s sys 0.006s
I20260812 19:13:06.759711 23074 raft_consensus.cc:3032] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:07.156369 23226 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
W20260812 19:13:06.777642 21323 consensus_meta.cc:287] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.648s user 0.010s sys 0.003s
I20260812 19:13:07.444733 21239 rpcz_store.cc:276] 0812 19:13:05.956303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:05.995099 (+ 38796us) service_pool.cc:227] Handling call
0812 19:13:06.034989 (+ 39890us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:06.157910 (+122921us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:06.157957 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:06.158014 (+ 57us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:06.158056 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:07.206377 (+1048321us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b5400303254. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:07.206436 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:07.206587 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":282054912}
I20260812 19:13:07.217070 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:07.445851 21244 rpcz_store.cc:269] 0812 19:13:05.966578 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:05.979916 (+ 13338us) service_pool.cc:227] Handling call
0812 19:13:07.445031 (+1465115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:06.688248 23148 raft_consensus.cc:3032] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:07.295660 21241 raft_consensus.cc:1226] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
W20260812 19:13:07.487383 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 1.40549e+06us
I20260812 19:13:07.519065 21324 raft_consensus.cc:1226] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:07.519364 21203 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
W20260812 19:13:07.126480 23164 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.879s user 0.000s sys 0.012s
I20260812 19:13:06.891980 22978 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:06.900146 23156 consensus_meta.cc:287] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.747s user 0.008s sys 0.006s
W20260812 19:13:07.018987 23176 consensus_meta.cc:287] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.038s user 0.008s sys 0.011s
I20260812 19:13:06.607375 22045 raft_consensus.cc:1226] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:06.659103 23173 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:07.490300 23228 raft_consensus.cc:1067] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Signalling peer 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:13:07.378845 23088 raft_consensus.cc:3032] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:06.688967 22048 raft_consensus.cc:1226] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:06.886511 23127 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:07.490824 23209 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:07.491338 23024 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:07.491999 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:07.626818 21321 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:07.622226 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:07.520440 23197 consensus_queue.cc:1046] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
I20260812 19:13:06.698841 23073 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:07.495922 23205 raft_consensus.cc:2726] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:06.563345 23062 raft_consensus.cc:2781] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:07.495499 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:07.019681 23171 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:07.490351 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:07.649639 21329 raft_consensus.cc:2454] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:13:07.650864 21200 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:07.651921 21329 raft_consensus.cc:1226] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:06.718508 23163 consensus_queue.cc:260] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:06.605660 23213 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:06.657949 23120 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:06.754346 23160 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:07.496793 23179 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:06.566541 23185 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:07.298539 23158 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
W20260812 19:13:07.443482 23240 log.cc:924] Time spent T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
W20260812 19:13:07.497891 23151 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.117s user 0.007s sys 0.012s
I20260812 19:13:06.758963 22896 consensus_queue.cc:1046] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.739s
W20260812 19:13:07.498098 22046 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40624 (request call id 295) took 1440 ms (client timeout 1730 ms). Trace:
I20260812 19:13:06.907552 23216 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:07.658409 23176 raft_consensus.cc:2781] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election won for term 3
W20260812 19:13:07.022491 23085 consensus_meta.cc:287] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.666s user 0.009s sys 0.001s
W20260812 19:13:07.126132 23143 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.876s user 0.000s sys 0.011s
I20260812 19:13:07.215786 23076 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:07.507328 21320 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
W20260812 19:13:07.081069 23206 thread.cc:636] tablet-download-pool [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.004s sys 0.006s
I20260812 19:13:06.622367 23175 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:06.563931 23199 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:07.319823 23217 raft_consensus.cc:3032] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:07.658073 23173 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:07.531518 21233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37894 (request call id 1438) took 1461 ms. Trace:
I20260812 19:13:07.532907 21327 raft_consensus.cc:1226] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:07.749171 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1316) took 1191 ms. Trace:
I20260812 19:13:07.502508 23200 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:07.456934 23082 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:07.755306 23258 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:07.742661 23201 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:07.755061 23082 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:07.755952 23258 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:07.756143 23201 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:06.563735 23183 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:07.759161 23258 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:07.760080 23173 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:07.378543 23109 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:07.763756 23173 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:07.500962 23074 raft_consensus.cc:729] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:07.746769 23076 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:07.765128 23074 consensus_queue.cc:260] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:07.507084 23116 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:07.767050 23116 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:07.758445 23201 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:07.580279 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:07.771155 23183 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:07.771936 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:07.773700 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:07.503139 23236 raft_consensus.cc:2726] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:07.758416 23082 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:07.514829 23148 raft_consensus.cc:729] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:07.504868 23243 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
W20260812 19:13:07.531771 23164 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.290s user 0.007s sys 0.017s
I20260812 19:13:07.505108 21323 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:07.515990 23169 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:07.590902 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1576) took 1750 ms. Trace:
I20260812 19:13:07.597288 21114 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
W20260812 19:13:07.598855 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 377933us
I20260812 19:13:07.600415 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:07.507921 23237 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:07.602375 21239 raft_consensus.cc:1226] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:07.746057 23170 raft_consensus.cc:3032] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:07.569888 23014 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:40598 (request call id 489) took 8586 ms. Trace:
I20260812 19:13:07.577288 23229 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:07.625531 23259 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:07.492542 23234 raft_consensus.cc:2781] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260812 19:13:07.630494 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 283554us
I20260812 19:13:06.562523 23184 consensus_queue.cc:260] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:07.503147 23226 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:07.595559 23088 raft_consensus.cc:729] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:07.623988 23132 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:07.640511 21321 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:07.621220 23204 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:07.645172 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:07.622747 23239 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:07.646173 21204 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.70.132:39579 timed out after 1.770s (SENT) [suppressed 15 similar messages]
I20260812 19:13:07.620103 22045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40614 (request call id 356) took 1803 ms. Trace:
I20260812 19:13:07.571821 22978 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:07.626240 23127 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:07.629644 23093 raft_consensus.cc:3032] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:07.791790 21324 raft_consensus.cc:1226] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
W20260812 19:13:07.713475 22046 rpcz_store.cc:269] 0812 19:13:06.057625 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:06.076558 (+ 18933us) service_pool.cc:227] Handling call
0812 19:13:07.497838 (+1421280us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b54004bf854. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:07.497991 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":467518208}
I20260812 19:13:07.653384 21200 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:07.888301 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 235031us
I20260812 19:13:07.888072 23239 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:07.888649 21238 raft_consensus.cc:1226] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:07.889005 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1324) took 1323 ms. Trace:
I20260812 19:13:07.889394 21324 rpcz_store.cc:276] 0812 19:13:06.565436 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:07.532560 (+967124us) service_pool.cc:227] Handling call
0812 19:13:07.791610 (+259050us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:07.887714 (+ 96104us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:07.888775 (+ 1061us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b5400230ad0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:07.888937 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":594758528}
I20260812 19:13:07.629345 23220 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:07.887495 23204 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:07.632966 23250 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:07.701718 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1325) took 1136 ms. Trace:
I20260812 19:13:07.702687 21236 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1326) took 1934 ms. Trace:
I20260812 19:13:07.703830 21241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1328) took 1889 ms. Trace:
I20260812 19:13:07.700775 21240 raft_consensus.cc:1226] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:07.576324 23156 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:07.626256 23209 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:07.648023 23188 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:07.018935 23208 consensus_meta.cc:287] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.596s user 0.003s sys 0.009s
I20260812 19:13:07.670514 23062 raft_consensus.cc:686] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:07.719465 21235 raft_consensus.cc:2454] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:07.726168 23185 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:07.726819 23143 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.478s user 0.006s sys 0.020s
I20260812 19:13:07.727195 23085 raft_consensus.cc:2726] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:07.719494 22056 raft_consensus.cc:1226] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:07.707262 23120 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:07.668426 23257 consensus_queue.cc:1046] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.085s
I20260812 19:13:07.731805 23225 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:07.733011 22896 raft_consensus.cc:3032] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:07.943496 23120 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:07.733669 23176 raft_consensus.cc:686] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:07.944773 23176 consensus_queue.cc:237] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:07.734158 23117 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:07.947069 23117 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:07.950253 23117 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:07.739391 21154 raft_consensus.cc:2454] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:07.721448 23216 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:07.742887 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1317) took 1184 ms. Trace:
I20260812 19:13:07.746737 21112 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:07.658735 23205 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:07.522424 21203 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:07.309432 23212 raft_consensus.cc:3032] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:07.712172 23158 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:07.742600 23203 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:07.749722 21327 rpcz_store.cc:276] 0812 19:13:06.557988 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:06.559704 (+ 1716us) service_pool.cc:227] Handling call
0812 19:13:07.532348 (+972644us) spinlock_profiling.cc:243] Waited 4.68 ms on lock 0x7b5800271898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:07.532627 (+ 279us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:07.748897 (+216270us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:07.749109 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10771712}
W20260812 19:13:07.744457 23206 thread.cc:602] tablet-download-pool [worker] (thread pool) Time spent starting thread: real 1.223s user 0.006s sys 0.031s
I20260812 19:13:07.748733 21233 rpcz_store.cc:276] 0812 19:13:06.070099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:06.331425 (+261326us) service_pool.cc:227] Handling call
0812 19:13:06.331879 (+ 454us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:07.206494 (+874615us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:07.531160 (+324666us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b54001518d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:07.531427 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":606312704}
I20260812 19:13:07.751303 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:07.753201 22052 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:07.748561 23217 raft_consensus.cc:729] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:07.780011 23243 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:07.994292 22047 raft_consensus.cc:2379] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 2: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:07.570369 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:07.716745 23163 raft_consensus.cc:3037] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:07.776130 23148 consensus_queue.cc:260] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:07.780136 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:07.594530 22012 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.848s user 0.004s sys 0.012s
I20260812 19:13:07.701373 23213 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:07.887342 23093 raft_consensus.cc:729] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:07.631546 23024 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:07.891811 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:07.857113 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:07.905771 21241 rpcz_store.cc:276] 0812 19:13:05.814822 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:05.815701 (+ 879us) service_pool.cc:227] Handling call
0812 19:13:05.846117 (+ 30416us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:07.527324 (+1681207us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:07.703559 (+176235us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b5400256cd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:07.703771 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3293429632}
I20260812 19:13:07.892434 21243 raft_consensus.cc:2379] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:07.907299 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:07.908401 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:08.003640 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 223724us
I20260812 19:13:08.004525 21324 raft_consensus.cc:2379] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 2: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:07.899617 22011 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:07.905836 21236 rpcz_store.cc:276] 0812 19:13:05.768515 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:05.768730 (+ 215us) service_pool.cc:227] Handling call
0812 19:13:05.769117 (+ 387us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:07.207459 (+1438342us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:07.702350 (+494891us) spinlock_profiling.cc:243] Waited 914 ms on lock 0x7b54000b5754. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:07.702587 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2102455168}
I20260812 19:13:07.905753 21328 rpcz_store.cc:276] 0812 19:13:06.565749 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:07.659795 (+1094046us) service_pool.cc:227] Handling call
0812 19:13:07.700941 (+ 41146us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:07.701243 (+ 302us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:07.701284 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:07.701344 (+ 60us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:07.701387 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:07.701465 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:07.701634 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:07.916810 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37894 (request call id 1433) took 1851 ms. Trace:
I20260812 19:13:07.779064 23116 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:08.005738 21240 rpcz_store.cc:276] 0812 19:13:06.065839 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:06.319561 (+253722us) service_pool.cc:227] Handling call
0812 19:13:06.340817 (+ 21256us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:07.907497 (+1566680us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:07.916504 (+ 9007us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b5400040954. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:07.916730 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3058918784}
I20260812 19:13:07.932823 23255 raft_consensus.cc:2781] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:07.906813 23132 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:07.791714 21318 rpcz_store.cc:276] 0812 19:13:05.840237 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:05.840408 (+ 171us) service_pool.cc:227] Handling call
0812 19:13:05.897127 (+ 56719us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:05.897358 (+ 231us) raft_consensus.cc:1520] Early marking committed up to index 2
0812 19:13:07.218454 (+1321096us) log.cc:842] Serialized 14 byte log entry
0812 19:13:07.590371 (+371917us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:07.590452 (+ 81us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:07.590513 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:07.590638 (+ 125us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:07.590818 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1469587,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":8374,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":150,"thread_start_us":1661111,"threads_started":2,"wal-append.queue_time_us":367419}
I20260812 19:13:07.766498 23074 raft_consensus.cc:3037] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
W20260812 19:13:07.934994 21235 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1429) took 1888 ms (client timeout 2061 ms). Trace:
I20260812 19:13:08.014786 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:07.935468 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 338331us
W20260812 19:13:08.014578 21235 rpcz_store.cc:269] 0812 19:13:06.046187 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:06.319254 (+273067us) service_pool.cc:227] Handling call
0812 19:13:07.934799 (+1615545us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b540018fe54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:07.934927 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2811031936}
I20260812 19:13:08.015797 21321 raft_consensus.cc:2379] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 2: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:08.016121 21114 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:07.777918 23199 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:07.906972 23183 raft_consensus.cc:2781] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:13:07.851238 23236 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:07.864070 23201 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:07.937805 23209 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:07.938082 23204 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:07.939293 21238 raft_consensus.cc:1226] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:07.780890 23246 consensus_queue.cc:1046] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.192s
I20260812 19:13:07.705286 23160 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:07.951035 23143 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:07.942399 22896 raft_consensus.cc:729] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:07.951624 22056 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 301) took 1182 ms. Trace:
I20260812 19:13:07.951987 21319 raft_consensus.cc:1226] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:07.953449 23257 raft_consensus.cc:3032] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:07.956074 23137 raft_consensus.cc:3032] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:07.956565 23226 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:07.959227 23156 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:07.962517 23127 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:07.936713 22978 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:07.890427 23239 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:07.781296 22048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40614 (request call id 361) took 1637 ms. Trace:
I20260812 19:13:07.982167 21320 rpcz_store.cc:276] 0812 19:13:06.558283 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:06.559797 (+ 1514us) service_pool.cc:227] Handling call
0812 19:13:07.742817 (+1183020us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:07.782269 23164 raft_consensus.cc:686] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:07.934482 23208 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:07.983472 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 337333us
W20260812 19:13:07.984910 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 238308us
I20260812 19:13:07.907426 23170 raft_consensus.cc:729] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:07.887511 23234 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:07.826259 21323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1580) took 1983 ms (client timeout 1922 ms). Trace:
W20260812 19:13:07.987782 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 79017.1us
I20260812 19:13:07.992453 21233 raft_consensus.cc:1226] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:07.913772 23250 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:07.759331 23196 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:07.999940 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:07.906898 23082 consensus_queue.cc:1046] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
I20260812 19:13:07.940676 23229 raft_consensus.cc:686] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:13:08.007329 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 107825us
I20260812 19:13:07.987429 23188 raft_consensus.cc:2726] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:07.986980 23212 raft_consensus.cc:729] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:13:08.012005 22012 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.783s user 0.005s sys 0.013s
I20260812 19:13:07.989531 23117 raft_consensus.cc:2781] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:08.014680 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:07.936650 23237 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:07.915005 23198 consensus_queue.cc:1046] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.077s
I20260812 19:13:07.991266 23205 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:07.907600 23259 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:08.631089 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 615068us
I20260812 19:13:08.679562 21329 raft_consensus.cc:2379] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 2: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:08.680011 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1346) took 1095 ms. Trace:
I20260812 19:13:08.030107 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:08.680534 21233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1349) took 1060 ms. Trace:
I20260812 19:13:08.681450 21327 raft_consensus.cc:2379] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:08.680037 21114 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:08.017700 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:08.680608 21238 rpcz_store.cc:276] 0812 19:13:07.584472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:07.937506 (+353034us) service_pool.cc:227] Handling call
0812 19:13:07.939017 (+ 1511us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:08.679730 (+740713us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:08.679939 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:08.643791 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:08.684248 21328 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:13:08.680949 21233 rpcz_store.cc:276] 0812 19:13:07.620683 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:07.991692 (+371009us) service_pool.cc:227] Handling call
0812 19:13:07.992179 (+ 487us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:08.680274 (+688095us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:08.680475 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:08.641389 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 1.11898e+06us
I20260812 19:13:08.686219 21203 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:08.165771 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:13:08.684674 21320 raft_consensus.cc:1226] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:07.891206 22045 rpcz_store.cc:276] 0812 19:13:05.817009 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:05.822145 (+ 5136us) service_pool.cc:227] Handling call
0812 19:13:06.606802 (+784657us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:06.607143 (+ 341us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:07.572277 (+965134us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:07.601143 (+ 28866us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35939072}
I20260812 19:13:07.909348 23014 rpcz_store.cc:276] 0812 19:12:58.939664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:58.939941 (+ 277us) service_pool.cc:227] Handling call
0812 19:12:59.466817 (+526876us) ts_tablet_manager.cc:928] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
0812 19:13:03.924153 (+4457336us) tablet_copy_client.cc:361] Tablet copy session begun
0812 19:13:05.579937 (+1655784us) tablet_metadata.cc:641] Metadata flushed
0812 19:13:05.604830 (+ 24893us) tablet_copy_client.cc:490] Wrote new tablet metadata
0812 19:13:07.210730 (+1605900us) tablet_copy_client.cc:411] Wrote new consensus metadata
0812 19:13:07.299126 (+ 88396us) tablet_replica.cc:205] Creating consensus instance
0812 19:13:07.487667 (+188541us) ts_tablet_manager.cc:983] Replica registered.
0812 19:13:07.518849 (+ 31182us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":848,"tablet-copy.queue_time_us":298823,"thread_start_us":317851,"threads_started":2}
I20260812 19:13:08.637036 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1353) took 1002 ms. Trace:
I20260812 19:13:08.692065 21318 raft_consensus.cc:2379] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:08.311959 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:08.694914 21203 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:08.696165 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1588) took 2618 ms. Trace:
I20260812 19:13:08.693246 22056 rpcz_store.cc:276] 0812 19:13:06.769317 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:06.769536 (+ 219us) service_pool.cc:227] Handling call
0812 19:13:06.770012 (+ 476us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:07.950757 (+1180745us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:07.951327 (+ 570us) spinlock_profiling.cc:243] Waited 946 ms on lock 0x7b540064c8d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:07.951553 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2176532864}
I20260812 19:13:08.700398 21319 rpcz_store.cc:276] 0812 19:13:06.077501 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:06.077641 (+ 140us) service_pool.cc:227] Handling call
0812 19:13:06.106806 (+ 29165us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:08.638181 (+2531375us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:08.695818 (+ 57637us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b540023e1d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:08.696078 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4213984128}
I20260812 19:13:08.698868 21243 rpcz_store.cc:276] 0812 19:13:07.634757 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:08.003765 (+369008us) service_pool.cc:227] Handling call
0812 19:13:08.636453 (+632688us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:08.636669 (+ 216us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:08.636698 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:08.636741 (+ 43us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:08.636774 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:08.636830 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:08.636957 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:08.206473 23023 tablet_bootstrap.cc:492] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:13:08.709971 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:08.261627 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:08.206740 23062 consensus_queue.cc:237] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:08.697369 23201 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:08.712584 23160 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:08.713555 23201 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:08.012544 23076 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:08.714586 21322 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:08.715286 23250 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:08.050369 23024 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:08.637162 23257 raft_consensus.cc:729] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:08.715756 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1328) took 1738 ms. Trace:
I20260812 19:13:08.716181 21322 rpcz_store.cc:276] 0812 19:13:06.977502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:07.515679 (+538177us) service_pool.cc:227] Handling call
0812 19:13:08.715560 (+1199881us) spinlock_profiling.cc:243] Waited 719 ms on lock 0x7b540016e9d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:08.715685 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1654317952}
I20260812 19:13:08.716495 23024 raft_consensus.cc:2726] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:08.716713 23257 consensus_queue.cc:260] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:08.707947 23176 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.742s user 0.026s sys 0.007s
W20260812 19:13:08.717924 23176 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.752s user 0.026s sys 0.007s
I20260812 19:13:08.050858 23093 consensus_queue.cc:260] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:08.013914 23109 raft_consensus.cc:2781] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:08.715792 22896 consensus_queue.cc:260] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:08.718816 23093 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:08.718943 23109 raft_consensus.cc:686] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:08.719262 23205 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:08.555778 23167 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:08.721244 21328 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f"
dest_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:51552
I20260812 19:13:08.722203 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:08.728684 22056 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:08.724500 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:08.725327 22053 raft_consensus.cc:2379] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 2: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:08.038784 23206 tablet_copy_client.cc:316] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Beginning tablet copy session from remote peer at address 127.19.70.129:36551
I20260812 19:13:08.062415 23132 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:08.726538 21114 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:08.051921 23213 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:08.630652 23148 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:08.030385 23175 raft_consensus.cc:2781] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:08.729377 21328 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
W20260812 19:13:08.729985 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 94308us
I20260812 19:13:08.050079 23186 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:08.013769 23255 raft_consensus.cc:686] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:13:08.725837 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 87428.7us
I20260812 19:13:08.163686 23216 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:08.279999 23169 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:08.762337 21200 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:08.732908 21237 raft_consensus.cc:1226] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:08.063522 23158 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:08.631724 22052 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40598 (request call id 531) took 2467 ms (client timeout 1770 ms). Trace:
I20260812 19:13:08.734372 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:08.734869 21155 raft_consensus.cc:1226] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:08.682029 23137 raft_consensus.cc:729] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:08.735178 21115 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:08.731556 23226 raft_consensus.cc:2781] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:08.298931 23173 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:08.048198 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:08.701506 23234 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:08.706383 23204 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:08.713096 23237 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:08.731818 23014 tablet_copy_client.cc:791] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:13:08.783421 22978 raft_consensus.cc:3032] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:08.783730 23014 tablet_copy_client.cc:655] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:13:08.783969 22978 raft_consensus.cc:729] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:08.782212 23204 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:08.785787 21237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37894 (request call id 1435) took 2719 ms. Trace:
I20260812 19:13:08.709517 23188 raft_consensus.cc:2726] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:08.760834 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:08.765066 21328 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:08.720630 23117 raft_consensus.cc:686] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:08.063819 23217 consensus_queue.cc:260] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:08.737671 23220 consensus_meta.cc:287] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.847s user 0.009s sys 0.003s
I20260812 19:13:08.745795 23243 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:08.772261 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:08.775506 23073 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:08.685425 21323 rpcz_store.cc:269] 0812 19:13:05.843273 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:05.846572 (+ 3299us) service_pool.cc:227] Handling call
0812 19:13:07.821185 (+1974613us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:08.697572 23236 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:08.746770 23246 consensus_queue.cc:1046] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260812 19:13:08.731164 22047 raft_consensus.cc:2454] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:13:08.780620 23109 consensus_queue.cc:237] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:08.691273 23143 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:08.816200 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 81107.1us
W20260812 19:13:08.878549 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 101737us
I20260812 19:13:08.760897 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:08.879601 21321 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:08.832427 21287 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.19.70.190:35311, user_credentials={real_user=slave}} blocked reactor thread for 45537.2us
I20260812 19:13:08.881664 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:08.846637 23127 raft_consensus.cc:2781] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:08.012089 23088 consensus_queue.cc:260] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:08.159273 23261 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:08.784010 23023 tablet_bootstrap.cc:492] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:13:08.766336 23225 raft_consensus.cc:2781] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:08.865839 23217 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:08.766407 22052 rpcz_store.cc:269] 0812 19:13:06.164275 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:06.221596 (+ 57321us) service_pool.cc:227] Handling call
0812 19:13:08.631380 (+2409784us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54006103d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:08.631590 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2705032448}
I20260812 19:13:08.738888 23239 consensus_queue.cc:1046] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20260812 19:13:08.744707 23024 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:08.739997 23205 consensus_queue.cc:1046] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.096s
I20260812 19:13:08.685559 22048 rpcz_store.cc:276] 0812 19:13:06.144150 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:06.149353 (+ 5203us) service_pool.cc:227] Handling call
0812 19:13:06.154850 (+ 5497us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:07.623941 (+1469091us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:07.745796 (+121855us) spinlock_profiling.cc:243] Waited 413 ms on lock 0x7b54006135d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:07.781175 (+ 35379us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":950775040}
I20260812 19:13:08.735404 23167 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:08.769343 23175 raft_consensus.cc:686] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:13:08.870121 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 143689us
I20260812 19:13:08.746555 23250 consensus_queue.cc:1046] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.111s
I20260812 19:13:08.877698 23208 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:08.765714 23213 consensus_queue.cc:1046] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.094s
I20260812 19:13:08.847005 23226 raft_consensus.cc:686] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:08.887208 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:08.955776 21155 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:08.784978 22978 consensus_queue.cc:260] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:08.885808 21328 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:08.852504 23209 raft_consensus.cc:2781] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:08.784978 23137 consensus_queue.cc:260] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:08.859714 23120 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:08.892736 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 153741us
I20260812 19:13:08.899971 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:08.975159 21151 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:08.974965 23120 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:08.975307 21112 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:08.905642 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:08.978931 21153 raft_consensus.cc:3037] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:08.923571 23262 raft_consensus.cc:3032] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:08.981712 23262 raft_consensus.cc:729] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:08.852326 21329 raft_consensus.cc:2454] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
W20260812 19:13:08.878583 23200 consensus_meta.cc:287] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.116s user 0.011s sys 0.002s
I20260812 19:13:08.748193 23170 consensus_queue.cc:260] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:08.982952 23262 consensus_queue.cc:260] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:08.983847 23170 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:08.984787 23262 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:08.910749 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:08.770231 23169 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:08.816818 23173 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:08.963703 23023 ts_tablet_manager.cc:1397] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 2.519s user 0.038s sys 0.069s
I20260812 19:13:08.914083 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:08.917178 21243 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:13:09.015728 21243 raft_consensus.cc:595] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Received request to transfer leadership to TS f741102bfe1441d1b453e5a838649059
I20260812 19:13:08.786172 21237 rpcz_store.cc:276] 0812 19:13:06.066518 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:06.319144 (+252626us) service_pool.cc:227] Handling call
0812 19:13:06.319421 (+ 277us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:08.764249 (+2444828us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:08.785463 (+ 21214us) spinlock_profiling.cc:243] Waited 2.41 s on lock 0x7b54002fe754. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:08.785691 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5540576000}
I20260812 19:13:08.922721 23280 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:08.924010 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:08.892201 23127 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:08.928751 23257 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:08.866494 23196 consensus_queue.cc:1046] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260812 19:13:08.817314 23186 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:08.786275 23212 consensus_queue.cc:260] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:08.863755 23073 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:08.939469 21114 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:08.957001 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:08.958024 21321 raft_consensus.cc:2454] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
W20260812 19:13:09.048413 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 109102us
I20260812 19:13:09.048938 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1600) took 1149 ms. Trace:
I20260812 19:13:08.919178 23183 raft_consensus.cc:686] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:13:08.960914 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 68336us
I20260812 19:13:08.964046 21328 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:09.048465 21151 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:08.968647 23137 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:08.961069 23209 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:08.970016 23175 consensus_queue.cc:237] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:08.973109 23208 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:08.856676 23164 consensus_queue.cc:237] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:08.917626 23225 raft_consensus.cc:686] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:09.000926 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:09.064495 21200 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:09.010960 21112 leader_election.cc:302] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:08.915172 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 193261us
I20260812 19:13:08.990584 21153 raft_consensus.cc:2454] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
W20260812 19:13:09.065932 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 55085.5us
I20260812 19:13:09.066139 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:09.067426 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:09.002676 21152 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:09.019389 23204 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:08.999770 23184 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:09.022840 23117 consensus_queue.cc:237] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:08.919803 23024 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:09.012398 23213 raft_consensus.cc:2726] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:09.052485 23200 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:08.986269 23116 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:09.050998 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:09.049274 21321 rpcz_store.cc:276] 0812 19:13:07.899696 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:08.016410 (+116714us) service_pool.cc:227] Handling call
0812 19:13:08.760669 (+744259us) spinlock_profiling.cc:243] Waited 1.85 ms on lock 0x7b5800271898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:13:09.048873 (+288204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4248832}
I20260812 19:13:09.053450 21114 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:09.054539 23246 consensus_queue.cc:1046] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:13:09.056619 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:09.027971 23127 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:09.055688 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:51552 (request call id 1599) took 1157 ms. Trace:
I20260812 19:13:09.087369 21328 rpcz_store.cc:276] 0812 19:13:07.899254 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:08.684774 (+785520us) service_pool.cc:227] Handling call
0812 19:13:09.050220 (+365446us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:09.061390 23212 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:08.869028 23229 consensus_queue.cc:237] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:09.044965 22978 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:09.071446 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:09.092829 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:08.985821 23273 consensus_queue.cc:1046] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.091s
I20260812 19:13:09.001543 23237 consensus_queue.cc:1046] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260812 19:13:09.048354 23205 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:09.074723 23160 raft_consensus.cc:2781] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:09.074358 21243 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:13:09.075767 23183 consensus_queue.cc:237] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:09.047135 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 117062us
I20260812 19:13:08.914325 23255 consensus_queue.cc:237] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:09.083209 21112 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:09.105199 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:09.048178 23199 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:09.083804 21235 raft_consensus.cc:1226] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:09.084177 21244 raft_consensus.cc:1226] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:09.080714 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:09.062242 21150 raft_consensus.cc:1226] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:09.106079 21240 raft_consensus.cc:2379] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 3: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:09.060863 23209 raft_consensus.cc:3037] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:09.108327 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:09.088225 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:09.089651 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:09.087369 21155 raft_consensus.cc:2454] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:09.116350 21239 raft_consensus.cc:2379] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 3: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:08.906744 23261 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:09.064544 21033 catalog_manager.cc:5495] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 1 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:09.093650 21233 raft_consensus.cc:2379] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 3: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:09.094065 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:09.097123 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:09.077364 22896 raft_consensus.cc:3037] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:09.099087 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:09.063005 23259 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:09.092928 23243 raft_consensus.cc:2726] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:09.114969 21237 raft_consensus.cc:2379] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 2: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:09.090931 23281 raft_consensus.cc:2781] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:09.117470 21318 raft_consensus.cc:2362] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4. Re-sending same reply.
I20260812 19:13:09.076679 23226 consensus_queue.cc:237] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:09.121569 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.070371 23208 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:09.123479 21149 raft_consensus.cc:2379] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:09.128326 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:09.122990 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:09.123270 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.134969 21155 raft_consensus.cc:2454] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:09.128326 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.130139 21112 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:09.135684 21237 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:13:09.135857 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:09.136323 21112 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:09.121526 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.136680 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.137034 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:09.137718 21155 raft_consensus.cc:2379] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 3: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:09.138338 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:09.074841 23234 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:09.139518 23279 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:09.133388 21323 raft_consensus.cc:1226] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:09.102960 23160 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:09.140439 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:09.140595 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.144440 21235 raft_consensus.cc:2379] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 2: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:09.144616 21240 raft_consensus.cc:2379] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 4: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:09.140756 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.109032 23278 raft_consensus.cc:3032] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:09.144932 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.090687 23116 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:09.132858 22011 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:09.145252 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.145545 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:09.108530 23143 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:09.146991 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:09.117959 23156 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:09.151623 22051 raft_consensus.cc:2362] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3. Re-sending same reply.
I20260812 19:13:09.117702 23213 raft_consensus.cc:2781] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:09.149397 21150 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
I20260812 19:13:09.152462 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:09.154498 21203 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:09.122146 23173 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:09.158057 22053 raft_consensus.cc:2379] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 2: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:09.158622 21200 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:09.140187 23132 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:09.132287 23281 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:09.106112 23220 consensus_queue.cc:1046] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260812 19:13:09.171434 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:09.046126 23280 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:09.113761 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:09.159745 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.143963 23246 raft_consensus.cc:2781] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:09.163779 22011 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:09.164832 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.166188 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:09.163864 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:09.166920 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:09.170437 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:09.192137 21149 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:13:09.147317 23160 raft_consensus.cc:3037] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:09.109638 23205 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:09.131506 23216 consensus_queue.cc:1046] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260812 19:13:09.147909 23278 raft_consensus.cc:729] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:09.185021 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:09.174594 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:09.184508 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:09.157801 23199 raft_consensus.cc:2781] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:09.188637 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:09.197598 22051 raft_consensus.cc:2454] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:13:09.153467 23093 raft_consensus.cc:2781] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:09.192731 22011 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:09.195452 21158 raft_consensus.cc:2379] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 2: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:09.196066 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:09.202646 21322 raft_consensus.cc:2379] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 2: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:09.192421 21328 raft_consensus.cc:2454] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:13:09.202982 22011 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:09.200542 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:09.203830 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:09.110245 23225 consensus_queue.cc:237] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:09.166467 23173 raft_consensus.cc:2781] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:09.167822 23213 raft_consensus.cc:686] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:09.159818 23283 raft_consensus.cc:2726] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:09.195091 23158 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:09.198576 23236 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:09.204840 23109 raft_consensus.cc:2781] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:09.172396 23281 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:09.231597 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:09.232043 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.232301 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.229408 23259 raft_consensus.cc:2781] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:09.173369 23132 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:09.200874 23199 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:09.232900 23109 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:09.238838 23213 consensus_queue.cc:237] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:09.234109 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.230115 23250 raft_consensus.cc:2781] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:09.228776 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:09.245855 21236 raft_consensus.cc:2379] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 2: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:09.279711 21149 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:09.248117 21240 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:09.252925 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:09.188740 23169 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:09.188376 21169 tablet_copy_service.cc:140] P f741102bfe1441d1b453e5a838649059: Received BeginTabletCopySession request for tablet c31c6a2accba4d29ba096500b9d1a911 from peer 63db10063a7d4eb2a9cc9dc952654706 ({username='slave'} at 127.0.0.1:51004)
I20260812 19:13:09.160147 23156 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:09.164598 23143 consensus_queue.cc:1046] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260812 19:13:09.237922 23216 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:09.265355 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:09.271214 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:09.273988 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:09.277256 23109 raft_consensus.cc:3037] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:09.333113 21324 raft_consensus.cc:2379] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:09.212764 23246 raft_consensus.cc:686] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:09.334579 23246 consensus_queue.cc:237] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:09.278951 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:09.242563 23277 raft_consensus.cc:3032] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:09.338989 23277 raft_consensus.cc:729] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:09.340418 21158 raft_consensus.cc:2454] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
I20260812 19:13:09.230361 23093 raft_consensus.cc:686] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:09.292689 21112 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:09.233309 21323 raft_consensus.cc:1226] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: [1.2-1.2]
I20260812 19:13:09.304921 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:09.305207 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 202211us
I20260812 19:13:09.243359 23073 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:09.308969 22008 leader_election.cc:334] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.703s (SENT)
W20260812 19:13:09.273293 23163 consensus_meta.cc:287] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 1.271s user 0.009s sys 0.005s
I20260812 19:13:09.255896 23259 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:09.309288 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.257310 23173 raft_consensus.cc:686] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:09.143224 23279 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:09.276115 23199 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:09.348675 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.373831 21240 raft_consensus.cc:2454] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
I20260812 19:13:09.352227 22008 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:09.232549 23158 raft_consensus.cc:504] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:09.238898 23261 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:09.278793 23250 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:09.307029 22048 raft_consensus.cc:1226] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:09.363503 21149 raft_consensus.cc:1261] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:13:09.314193 23024 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:09.305024 23132 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:09.367031 21239 raft_consensus.cc:2379] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 2: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:09.340372 23277 consensus_queue.cc:260] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:09.370848 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:08.986330 23062 consensus_queue.cc:1159] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d1e1da0b9585460e8c365bf55679d4fa" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260812 19:13:09.344097 21169 tablet_copy_service.cc:161] P f741102bfe1441d1b453e5a838649059: Beginning new tablet copy session on tablet c31c6a2accba4d29ba096500b9d1a911 from peer 63db10063a7d4eb2a9cc9dc952654706 at {username='slave'} at 127.0.0.1:51004: session id = 63db10063a7d4eb2a9cc9dc952654706-c31c6a2accba4d29ba096500b9d1a911
I20260812 19:13:09.242497 23023 raft_consensus.cc:348] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:09.279559 23278 consensus_queue.cc:260] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:09.340631 23109 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:09.401839 21320 raft_consensus.cc:1226] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:09.373835 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:09.697693 21203 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.015s sys 0.018s
W20260812 19:13:09.698264 21203 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.533s user 0.015s sys 0.018s
I20260812 19:13:09.427755 21157 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:09.377687 21204 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:09.692649 22051 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:13:09.374187 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:09.773149 23278 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:09.775694 23281 consensus_meta.cc:287] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.510s user 0.011s sys 0.002s
W20260812 19:13:09.776469 22978 consensus_meta.cc:287] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.676s user 0.010s sys 0.000s
I20260812 19:13:09.687551 23220 raft_consensus.cc:2781] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:09.440757 23284 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:09.401952 23208 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:09.792900 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 415338us
I20260812 19:13:09.793401 21150 raft_consensus.cc:2454] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:09.793998 21204 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:09.588130 23093 consensus_queue.cc:237] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:09.352131 23216 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:09.664153 23280 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:09.600267 23117 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.006s sys 0.027s
W20260812 19:13:09.376185 23074 consensus_meta.cc:287] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 1.346s user 0.014s sys 0.000s
W20260812 19:13:09.378400 23203 consensus_meta.cc:287] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.387s user 0.011s sys 0.003s
I20260812 19:13:09.782006 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.815469 21239 raft_consensus.cc:2379] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 2: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:09.368525 23259 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:09.393750 23148 consensus_meta.cc:287] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.632s user 0.008s sys 0.004s
I20260812 19:13:09.790356 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.372418 23258 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:09.798064 23196 raft_consensus.cc:2726] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:09.835367 23216 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:09.799489 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:09.702450 23250 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:09.808375 21204 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:09.441205 23234 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:09.693059 23023 raft_consensus.cc:729] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:13:09.842937 21243 raft_consensus.cc:595] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:13:09.843649 21243 raft_consensus.cc:603] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:13:09.767418 23261 raft_consensus.cc:2726] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:09.792289 23281 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:09.792541 23220 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:09.813170 23160 consensus_meta.cc:287] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.608s user 0.008s sys 0.002s
W20260812 19:13:09.814011 23212 consensus_meta.cc:287] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.716s user 0.010s sys 0.000s
I20260812 19:13:09.784363 23254 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:09.816825 23158 leader_election.cc:288] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:09.813439 23284 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:09.601123 23163 consensus_queue.cc:1046] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.211s
W20260812 19:13:09.809719 23117 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.723s user 0.006s sys 0.029s
I20260812 19:13:09.824738 21327 raft_consensus.cc:1261] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260812 19:13:09.821599 23137 consensus_meta.cc:287] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.766s user 0.011s sys 0.001s
I20260812 19:13:09.871915 23220 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:09.831045 23257 consensus_meta.cc:287] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.803s user 0.010s sys 0.002s
I20260812 19:13:09.810462 23156 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:09.821765 23170 consensus_meta.cc:287] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.837s user 0.013s sys 0.000s
I20260812 19:13:09.900861 23284 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:09.832185 23208 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:09.887102 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.871672 23196 raft_consensus.cc:2781] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:09.910764 22053 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40624 (request call id 308) took 1027 ms. Trace:
I20260812 19:13:09.811573 23283 raft_consensus.cc:2781] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:09.891785 21237 raft_consensus.cc:2379] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:09.600778 23173 consensus_queue.cc:237] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:09.821434 23109 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:09.894434 21328 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:09.837357 23204 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:09.844121 23276 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:09.902275 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:09.875033 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.905181 21151 raft_consensus.cc:2396] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 2 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260812 19:13:09.902464 21240 raft_consensus.cc:2454] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
I20260812 19:13:09.906178 23167 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:09.901923 23088 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:09.955590 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.911365 22053 rpcz_store.cc:276] 0812 19:13:08.883058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:08.883254 (+ 196us) service_pool.cc:227] Handling call
0812 19:13:09.910375 (+1027121us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54004bf854. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:09.910667 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":395139456}
I20260812 19:13:09.895200 23239 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:09.906653 23279 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:09.944770 23217 consensus_meta.cc:287] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.030s user 0.013s sys 0.000s
I20260812 19:13:09.965932 21239 raft_consensus.cc:1226] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:09.968742 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:09.970643 21240 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:09.887248 23160 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:09.950599 22050 raft_consensus.cc:1226] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:09.975762 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:09.965574 23229 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.723s user 0.052s sys 0.134s
W20260812 19:13:09.970021 23127 consensus_meta.cc:287] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.870s user 0.009s sys 0.005s
I20260812 19:13:09.999684 23127 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:09.917641 23281 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:09.945811 22896 consensus_meta.cc:287] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.824s user 0.013s sys 0.000s
I20260812 19:13:09.917984 23196 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
W20260812 19:13:09.974186 21200 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.814s user 0.009s sys 0.027s
W20260812 19:13:10.010993 21200 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.851s user 0.009s sys 0.027s
W20260812 19:13:10.011739 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 853191us
I20260812 19:13:09.844812 23258 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:09.972030 21287 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.916s user 0.014s sys 0.022s
I20260812 19:13:10.001672 21155 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
W20260812 19:13:09.969175 23246 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.016s sys 0.020s
W20260812 19:13:10.015810 21287 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.960s user 0.014s sys 0.022s
I20260812 19:13:09.918412 23208 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:10.018479 21200 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:09.997582 23229 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.755s user 0.053s sys 0.134s
I20260812 19:13:09.892565 23254 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:10.028990 21153 raft_consensus.cc:2454] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:10.039311 21203 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:09.950644 23204 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:10.046136 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:10.050216 21203 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:10.048188 21115 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:09.995967 23283 raft_consensus.cc:686] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:09.969566 23239 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:10.017932 23246 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.680s user 0.017s sys 0.021s
I20260812 19:13:10.014209 23196 raft_consensus.cc:3037] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
W20260812 19:13:10.057632 21287 connection.cc:232] RPC call timeout handler was delayed by -2.9446969338459894s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1525s for a response
W20260812 19:13:10.115542 21287 connection.cc:232] RPC call timeout handler was delayed by -2.9122969339368865s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15610000000000002s for a response
W20260812 19:13:10.116993 21287 connection.cc:232] RPC call timeout handler was delayed by -2.84749693388585s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1633s for a response
I20260812 19:13:10.046937 21154 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
W20260812 19:13:10.117391 21287 connection.cc:232] RPC call timeout handler was delayed by -2.5828969338908792s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1927s for a response
W20260812 19:13:10.117790 21287 connection.cc:232] RPC call timeout handler was delayed by -2.4622969338670373s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2061s for a response
I20260812 19:13:10.118602 21287 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:10.085007 23076 raft_consensus.cc:3032] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:10.087086 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:10.088882 23236 raft_consensus.cc:3032] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:10.120074 21319 raft_consensus.cc:2362] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4. Re-sending same reply.
I20260812 19:13:10.124825 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1610) took 1045 ms. Trace:
I20260812 19:13:10.125245 21319 rpcz_store.cc:276] 0812 19:13:09.079508 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.079650 (+ 142us) service_pool.cc:227] Handling call
0812 19:13:10.086798 (+1007148us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b5800271898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:13:10.124732 (+ 37934us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3589504}
I20260812 19:13:10.101771 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34764 (request call id 16) took 1046 ms. Trace:
I20260812 19:13:10.126821 21243 rpcz_store.cc:276] 0812 19:13:09.055161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.056963 (+ 1802us) service_pool.cc:227] Handling call
0812 19:13:10.101452 (+1044489us) spinlock_profiling.cc:243] Waited 742 ms on lock 0x7b5400043dd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa0bc3 00007f52de9ea81a 00007f52d80e9d04 00007f52d80e9c92 00007f52d80e9bc1 00007f52d80e9b3c 00007f52d80e8df2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:10.101626 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1706516352}
I20260812 19:13:10.105155 22048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 537) took 1031 ms. Trace:
I20260812 19:13:10.107076 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:09.907608 23023 consensus_queue.cc:260] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:10.148876 21287 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:10.150941 21287 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:10.129061 23292 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:10.152801 21287 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:10.129990 23073 raft_consensus.cc:2726] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:13:10.131219 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:10.135900 23024 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:10.116180 23148 raft_consensus.cc:2781] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:10.141090 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:10.144001 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1605) took 1393 ms. Trace:
I20260812 19:13:10.111692 23156 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:10.146024 23082 raft_consensus.cc:3032] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:10.185000 23156 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:10.112658 23205 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:10.123150 23076 raft_consensus.cc:729] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:10.055069 23281 consensus_queue.cc:1046] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.105s
I20260812 19:13:10.122886 23236 raft_consensus.cc:729] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:10.182092 22056 raft_consensus.cc:2379] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 2: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:09.974645 23276 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:10.053220 23293 raft_consensus.cc:2781] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:10.111881 23288 raft_consensus.cc:2781] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:10.192534 22056 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40598 (request call id 533) took 1126 ms. Trace:
I20260812 19:13:10.206914 22056 rpcz_store.cc:276] 0812 19:13:09.065870 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.066065 (+ 195us) service_pool.cc:227] Handling call
0812 19:13:10.140846 (+1074781us) spinlock_profiling.cc:243] Waited 3.71 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:13:10.192441 (+ 51595us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8539392}
I20260812 19:13:10.185863 23283 consensus_queue.cc:237] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:10.185016 23082 raft_consensus.cc:729] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:10.155052 23137 raft_consensus.cc:2726] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:10.224818 23282 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:10.228040 23282 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:10.164880 23297 raft_consensus.cc:2726] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:10.235966 23176 raft_consensus.cc:3032] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:10.236733 23176 raft_consensus.cc:729] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:10.210639 23288 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:10.193339 23292 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:10.136034 21152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54796 (request call id 1542) took 1176 ms. Trace:
I20260812 19:13:10.148758 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:10.139029 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37894 (request call id 1443) took 1056 ms. Trace:
I20260812 19:13:10.181057 21327 rpcz_store.cc:276] 0812 19:13:08.750097 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:08.752475 (+ 2378us) service_pool.cc:227] Handling call
0812 19:13:08.752784 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:09.891741 (+1138957us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:10.143597 (+251856us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400312ed4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:10.143903 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2448857856}
I20260812 19:13:10.141683 22048 rpcz_store.cc:276] 0812 19:13:09.073846 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.205138 (+131292us) service_pool.cc:227] Handling call
0812 19:13:09.231197 (+ 26059us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:09.765877 (+534680us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:10.104699 (+338822us) spinlock_profiling.cc:243] Waited 34.3 ms on lock 0x7b5400452254. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:10.105044 (+ 345us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":78886272}
I20260812 19:13:10.162003 23074 raft_consensus.cc:2726] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:13:10.202649 23076 consensus_queue.cc:260] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:10.224704 23120 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:10.259079 23198 raft_consensus.cc:3032] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:10.166766 23143 raft_consensus.cc:2726] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:13:10.190367 23148 raft_consensus.cc:686] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:10.651538 23082 consensus_queue.cc:260] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:10.653810 23082 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:10.255347 23257 raft_consensus.cc:3032] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:10.654596 23148 consensus_queue.cc:237] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:10.656515 23257 raft_consensus.cc:729] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:13:10.323810 23262 consensus_meta.cc:287] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 1.338s user 0.009s sys 0.009s
I20260812 19:13:10.210947 23127 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:10.208684 23274 raft_consensus.cc:3032] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Stepping down as leader of term 2
W20260812 19:13:10.316789 23209 consensus_meta.cc:287] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.194s user 0.013s sys 0.002s
W20260812 19:13:10.416718 23259 consensus_meta.cc:287] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.580s user 0.004s sys 0.009s
I20260812 19:13:10.261278 23204 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:10.321714 23220 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:10.266256 23239 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:10.621605 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:10.628420 21244 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:10.193655 22045 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:10.369949 23199 consensus_meta.cc:287] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.974s user 0.012s sys 0.001s
I20260812 19:13:10.165661 23160 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:10.566461 21243 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49361681c5c1421594d10cd52061ec44"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:13:10.629390 21238 rpcz_store.cc:276] 0812 19:13:09.082877 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.082995 (+ 118us) service_pool.cc:227] Handling call
0812 19:13:09.086574 (+ 3579us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:09.094041 (+ 7467us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:09.094083 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:09.094129 (+ 46us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:09.094155 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:10.138680 (+1044525us) spinlock_profiling.cc:243] Waited 7.26 ms on lock 0x7b54002c3dd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:10.138744 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:10.138945 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16705408}
I20260812 19:13:10.680925 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:10.681154 21243 raft_consensus.cc:595] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Received request to transfer leadership to TS f741102bfe1441d1b453e5a838649059
I20260812 19:13:10.241755 23116 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:10.258627 23243 raft_consensus.cc:3032] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:10.682479 23243 raft_consensus.cc:729] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:10.683128 23143 raft_consensus.cc:3032] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:10.683694 22050 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 539) took 1588 ms. Trace:
I20260812 19:13:10.187264 23024 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:10.296657 23236 consensus_queue.cc:260] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:10.262912 23186 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:10.630810 21152 rpcz_store.cc:276] 0812 19:13:08.959669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.339294 (+379625us) service_pool.cc:227] Handling call
0812 19:13:09.339642 (+ 348us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:09.822294 (+482652us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:09.822341 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:09.822397 (+ 56us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:09.822438 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:10.135687 (+313249us) spinlock_profiling.cc:243] Waited 482 ms on lock 0x7b540030d9d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:10.135753 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:10.135945 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1109435264}
I20260812 19:13:10.163228 22046 raft_consensus.cc:2362] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2. Re-sending same reply.
I20260812 19:13:10.692523 23199 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:10.693871 22046 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40614 (request call id 369) took 1598 ms (client timeout 1738 ms). Trace:
W20260812 19:13:10.419804 23278 consensus_meta.cc:287] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.610s user 0.012s sys 0.002s
W20260812 19:13:10.694451 22046 rpcz_store.cc:269] 0812 19:13:09.095329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.109493 (+ 14164us) service_pool.cc:227] Handling call
0812 19:13:10.106810 (+997317us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:13:10.693615 (+586805us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23003008}
W20260812 19:13:10.695427 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 472228us
I20260812 19:13:10.695705 23201 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:10.696623 23201 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:10.300058 23254 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:10.340237 23282 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:10.333552 23276 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:10.229328 23293 raft_consensus.cc:686] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:10.702777 23198 raft_consensus.cc:729] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:10.703301 21243 raft_consensus.cc:603] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:13:10.630333 23117 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:10.669309 23288 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
W20260812 19:13:10.565208 21156 consensus_meta.cc:287] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.666s user 0.008s sys 0.004s
I20260812 19:13:10.313153 23277 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:10.322738 23208 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:10.660364 23269 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:10.675521 23156 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:10.257983 23258 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:10.676883 21157 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1631) took 1794 ms (client timeout 1574 ms). Trace:
I20260812 19:13:10.679491 23259 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:10.723934 21244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1453) took 1621 ms (client timeout 1510 ms). Trace:
W20260812 19:13:10.724432 21244 rpcz_store.cc:269] 0812 19:13:09.102174 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.121239 (+ 19065us) service_pool.cc:227] Handling call
0812 19:13:10.723482 (+1602243us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b5400043dd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:10.723760 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3325203584}
I20260812 19:13:10.725368 23024 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:10.725986 23259 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:10.676921 23023 ts_tablet_manager.cc:1428] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 1.631s user 0.018s sys 0.017s
W20260812 19:13:10.425055 23250 consensus_meta.cc:287] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.538s user 0.004s sys 0.008s
W20260812 19:13:10.726751 23023 ts_tablet_manager.cc:1460] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: Tablet startup took 1711ms
W20260812 19:13:10.722278 21157 rpcz_store.cc:269] 0812 19:13:08.881817 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:08.896470 (+ 14653us) service_pool.cc:227] Handling call
0812 19:13:10.676236 (+1779766us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b5400146ed4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:10.676531 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":948142464}
I20260812 19:13:10.727262 23250 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:10.173544 23200 consensus_meta.cc:287] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.082s user 0.008s sys 0.006s
I20260812 19:13:10.684281 22050 rpcz_store.cc:276] 0812 19:13:09.094937 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.231326 (+136389us) service_pool.cc:227] Handling call
0812 19:13:09.918689 (+687363us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:09.993771 (+ 75082us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:10.683048 (+689277us) spinlock_profiling.cc:243] Waited 679 ms on lock 0x7b54004bf850. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:10.683390 (+ 342us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1561320192}
I20260812 19:13:10.750399 21244 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:13:10.684598 23243 consensus_queue.cc:260] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:10.683743 23143 raft_consensus.cc:729] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:10.690511 23236 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:10.691295 23274 raft_consensus.cc:729] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:10.691476 22047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40614 (request call id 363) took 1644 ms. Trace:
I20260812 19:13:10.694594 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:10.719254 23292 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:10.723299 23076 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:10.677829 23209 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:10.728353 21151 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1549) took 1765 ms (client timeout 1594 ms). Trace:
I20260812 19:13:10.242136 23176 consensus_queue.cc:260] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:10.754953 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:10.710421 23198 consensus_queue.cc:260] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:10.749316 21156 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:10.766314 21149 raft_consensus.cc:2379] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:13:10.771629 21155 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51004 (request call id 182) took 1628 ms (client timeout 1775 ms). Trace:
I20260812 19:13:10.721645 23269 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:10.780417 21154 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1619) took 2070 ms (client timeout 1580 ms). Trace:
W20260812 19:13:10.780210 21155 rpcz_store.cc:269] 0812 19:13:09.141392 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.152137 (+ 10745us) service_pool.cc:227] Handling call
0812 19:13:10.769033 (+1616896us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b5400195ad4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:10.769322 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1945526016}
W20260812 19:13:10.781131 21154 rpcz_store.cc:269] 0812 19:13:08.709549 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:08.709707 (+ 158us) service_pool.cc:227] Handling call
0812 19:13:10.774997 (+2065290us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400182254. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:10.780201 (+ 5204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2913655552}
I20260812 19:13:10.782279 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:10.786150 21152 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
W20260812 19:13:10.770905 21151 rpcz_store.cc:269] 0812 19:13:08.963158 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.050735 (+ 87577us) service_pool.cc:227] Handling call
0812 19:13:10.727860 (+1677125us) spinlock_profiling.cc:243] Waited 819 ms on lock 0x7b5400215054. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:10.728118 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1882883840}
I20260812 19:13:10.773388 21244 raft_consensus.cc:595] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:13:10.747210 23293 consensus_queue.cc:237] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:10.790253 21153 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1551) took 1760 ms (client timeout 1738 ms). Trace:
I20260812 19:13:10.791613 23250 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:10.793851 22055 raft_consensus.cc:1226] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 3. Ops: []
I20260812 19:13:10.792881 21244 raft_consensus.cc:603] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:13:10.797784 22055 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 534) took 1723 ms. Trace:
I20260812 19:13:10.798089 21150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1553) took 1660 ms. Trace:
I20260812 19:13:10.799127 21239 raft_consensus.cc:1226] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:10.798641 21150 rpcz_store.cc:276] 0812 19:13:09.094601 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.201581 (+106980us) service_pool.cc:227] Handling call
0812 19:13:10.754451 (+1552870us) spinlock_profiling.cc:243] Waited 546 ms on lock 0x7b5400215a54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:10.754945 (+ 494us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1255313792}
I20260812 19:13:10.801422 21155 raft_consensus.cc:2379] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 4: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:13:10.792155 21153 rpcz_store.cc:269] 0812 19:13:09.025690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.067186 (+ 41496us) service_pool.cc:227] Handling call
0812 19:13:10.783028 (+1715842us) spinlock_profiling.cc:243] Waited 946 ms on lock 0x7b54000c3854. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:10.786157 (+ 3129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2176931712}
I20260812 19:13:10.805651 21153 raft_consensus.cc:1226] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:10.808406 21153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54796 (request call id 1555) took 1655 ms. Trace:
I20260812 19:13:10.798370 22055 rpcz_store.cc:276] 0812 19:13:09.071998 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.072160 (+ 162us) service_pool.cc:227] Handling call
0812 19:13:10.762818 (+1690658us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:10.764719 (+ 1901us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:10.794475 (+ 29756us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:10.795807 (+ 1332us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26897280}
I20260812 19:13:10.809161 21153 rpcz_store.cc:276] 0812 19:13:09.151357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:10.804562 (+1653205us) service_pool.cc:227] Handling call
0812 19:13:10.805277 (+ 715us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:10.806299 (+ 1022us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:10.806574 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:10.782544 23269 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:10.765082 22047 rpcz_store.cc:276] 0812 19:13:09.049105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.050446 (+ 1341us) service_pool.cc:227] Handling call
0812 19:13:10.606326 (+1555880us) spinlock_profiling.cc:243] Waited 2.99 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52bce 000000000034aea9 00007f52de972a30 00007f52dea387b4 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:13:10.691338 (+ 85012us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6886144}
I20260812 19:13:10.816045 21155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1564) took 1026 ms. Trace:
I20260812 19:13:10.737865 23257 consensus_queue.cc:260] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:10.807849 23200 raft_consensus.cc:2726] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:10.833940 21155 rpcz_store.cc:276] 0812 19:13:09.784893 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:10.781774 (+996881us) service_pool.cc:227] Handling call
0812 19:13:10.811132 (+ 29358us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:10.836377 23198 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:10.844029 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260812 19:13:10.851667 23199 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:10.853781 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:13:10.854544 23176 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:10.837303 21153 raft_consensus.cc:1226] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 1. Current term is 2. Ops: []
W20260812 19:13:10.849426 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 88605.7us
I20260812 19:13:10.858984 21157 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:10.860791 21151 raft_consensus.cc:1226] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:10.861613 21151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54796 (request call id 1554) took 1738 ms. Trace:
W20260812 19:13:10.863445 23196 consensus_meta.cc:287] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.718s user 0.011s sys 0.007s
I20260812 19:13:10.863826 23220 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:10.864090 23196 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:10.865897 23243 raft_consensus.cc:3037] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:10.866146 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1366) took 1489 ms. Trace:
I20260812 19:13:10.866781 21239 rpcz_store.cc:276] 0812 19:13:09.377086 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.973439 (+596353us) service_pool.cc:227] Handling call
0812 19:13:09.992327 (+ 18888us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:10.802214 (+809887us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:10.865332 (+ 63118us) spinlock_profiling.cc:243] Waited 804 ms on lock 0x7b54001f8954. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:10.865915 (+ 583us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1850183424}
I20260812 19:13:10.867683 21152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 178) took 1772 ms. Trace:
I20260812 19:13:10.815038 23117 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:10.868167 21152 rpcz_store.cc:276] 0812 19:13:09.095750 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:10.802420 (+1706670us) service_pool.cc:227] Handling call
0812 19:13:10.866881 (+ 64461us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:10.867251 (+ 370us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:10.867293 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:10.867355 (+ 62us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:10.867386 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:10.867447 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:10.867594 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:10.870256 21152 raft_consensus.cc:1226] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:10.862001 21151 rpcz_store.cc:276] 0812 19:13:09.123631 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:10.815518 (+1691887us) service_pool.cc:227] Handling call
0812 19:13:10.860446 (+ 44928us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:10.861329 (+ 883us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:10.861553 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:10.873123 23117 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:10.874485 21156 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1625) took 2142 ms (client timeout 1930 ms). Trace:
W20260812 19:13:10.727281 23023 ts_tablet_manager.cc:1462] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: Trace:
0812 19:12:59.238181 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:59.238451 (+ 270us) service_pool.cc:227] Handling call
0812 19:12:59.575042 (+336591us) ts_tablet_manager.cc:928] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551)
0812 19:13:01.849084 (+2274042us) tablet_copy_client.cc:361] Tablet copy session begun
0812 19:13:03.066321 (+1217237us) tablet_metadata.cc:641] Metadata flushed
0812 19:13:03.066729 (+ 408us) tablet_copy_client.cc:490] Wrote new tablet metadata
0812 19:13:03.527827 (+461098us) tablet_copy_client.cc:411] Wrote new consensus metadata
0812 19:13:03.591666 (+ 63839us) tablet_replica.cc:205] Creating consensus instance
0812 19:13:03.598418 (+ 6752us) ts_tablet_manager.cc:983] Replica registered.
0812 19:13:03.598541 (+ 123us) inbound_call.cc:173] Queueing success response
0812 19:13:06.444721 (+2846180us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0812 19:13:09.051916 (+2607195us) ts_tablet_manager.cc:1429] Starting tablet replica
0812 19:13:09.239579 (+187663us) tablet_replica.cc:259] Starting instrumentation
0812 19:13:09.240436 (+ 857us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":5980,"mutex_wait_us":356,"raft.queue_time_us":7136,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":185,"tablet-copy.queue_time_us":173758,"tablet-download-pool.queue_time_us":265,"tablet-download-pool.run_cpu_time_us":20803,"tablet-download-pool.run_wall_time_us":1051014,"thread_start_us":192222,"threads_started":2}
I20260812 19:13:10.858932 21153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54796 (request call id 1557) took 1629 ms. Trace:
I20260812 19:13:10.875885 23257 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:10.847767 23143 consensus_queue.cc:260] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:10.837571 23274 consensus_queue.cc:260] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:10.896265 21152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54796 (request call id 1563) took 1195 ms. Trace:
I20260812 19:13:10.915022 21152 rpcz_store.cc:276] 0812 19:13:09.687178 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:10.869336 (+1182158us) service_pool.cc:227] Handling call
0812 19:13:10.869930 (+ 594us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:10.882413 (+ 12483us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:10.882739 (+ 326us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:10.896798 21153 rpcz_store.cc:276] 0812 19:13:09.228278 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:10.835938 (+1607660us) service_pool.cc:227] Handling call
0812 19:13:10.836747 (+ 809us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:10.856708 (+ 19961us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:10.857623 (+ 915us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:10.884225 21156 rpcz_store.cc:269] 0812 19:13:08.732145 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:08.732329 (+ 184us) service_pool.cc:227] Handling call
0812 19:13:10.874182 (+2141853us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400214654. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:10.874373 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2485686016}
I20260812 19:13:10.900635 21155 raft_consensus.cc:1261] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:13:10.903827 21151 raft_consensus.cc:1226] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:10.929066 23143 raft_consensus.cc:3037] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:10.930884 21152 raft_consensus.cc:1226] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:10.931396 23274 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:10.917699 21155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 187) took 1685 ms. Trace:
I20260812 19:13:10.932355 21155 rpcz_store.cc:276] 0812 19:13:09.232764 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:10.876698 (+1643934us) service_pool.cc:227] Handling call
0812 19:13:10.900021 (+ 23323us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:10.917333 (+ 17312us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:10.917624 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:10.934844 21151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1644) took 1040 ms. Trace:
I20260812 19:13:10.935410 21151 rpcz_store.cc:276] 0812 19:13:09.895042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:10.902712 (+1007670us) service_pool.cc:227] Handling call
0812 19:13:10.903454 (+ 742us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:10.934433 (+ 30979us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:10.934759 (+ 326us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:10.878660 23201 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:10.952448 23196 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:10.968772 21288 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.606s user 0.000s sys 0.027s
W20260812 19:13:10.973909 21203 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.898s user 0.011s sys 0.016s
W20260812 19:13:10.977108 21288 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.616s user 0.000s sys 0.028s
I20260812 19:13:10.969092 23220 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:10.980175 21203 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.904s user 0.012s sys 0.016s
W20260812 19:13:10.988570 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 1.62301e+06us
I20260812 19:13:10.990777 23199 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:11.005769 23254 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:11.006968 21115 leader_election.cc:334] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Timed out: RequestConsensusVote RPC to 127.19.70.132:39579 timed out after 1.730s (SENT)
I20260812 19:13:11.006857 23254 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:11.008086 21115 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:11.009768 23117 raft_consensus.cc:2726] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 19:13:11.015484 22008 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.260s user 0.025s sys 0.066s
W20260812 19:13:11.014320 21112 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.605s user 0.030s sys 0.055s
W20260812 19:13:11.005623 23175 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.398s user 0.016s sys 0.052s
W20260812 19:13:11.032699 23175 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.432s user 0.017s sys 0.052s
W20260812 19:13:11.033803 22008 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.342s user 0.026s sys 0.066s
W20260812 19:13:11.039220 21112 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.693s user 0.032s sys 0.059s
I20260812 19:13:11.032052 23254 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:10.987052 23201 consensus_queue.cc:1046] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20260812 19:13:11.006146 23199 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:11.035833 23226 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.178s user 0.019s sys 0.031s
W20260812 19:13:11.043488 22010 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.540s user 0.038s sys 0.024s
W20260812 19:13:11.046274 21114 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.363s user 0.018s sys 0.042s
W20260812 19:13:11.013597 23183 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.164s user 0.017s sys 0.021s
W20260812 19:13:11.033680 23255 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.584s user 0.017s sys 0.035s
I20260812 19:13:11.016394 23288 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:11.060061 23255 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.689s user 0.017s sys 0.036s
I20260812 19:13:11.060895 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.061396 21150 raft_consensus.cc:1226] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:11.062146 22047 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
W20260812 19:13:11.062484 23093 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.778s user 0.010s sys 0.015s
W20260812 19:13:11.063285 23093 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.832s user 0.010s sys 0.015s
I20260812 19:13:11.063726 21157 raft_consensus.cc:1226] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:11.013777 23209 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:11.064419 22010 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.643s user 0.038s sys 0.024s
W20260812 19:13:11.065598 21114 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.465s user 0.018s sys 0.043s
W20260812 19:13:11.070712 22011 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.224s user 0.017s sys 0.013s
W20260812 19:13:11.071249 22011 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.377s user 0.018s sys 0.013s
W20260812 19:13:11.073545 23226 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.294s user 0.020s sys 0.031s
W20260812 19:13:11.062410 21200 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.931s user 0.009s sys 0.023s
I20260812 19:13:11.077545 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:11.081861 21200 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.026s user 0.010s sys 0.023s
I20260812 19:13:11.071274 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:11.083408 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:11.084537 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 1.79177e+06us
I20260812 19:13:11.089854 23288 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:11.081449 21151 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
W20260812 19:13:11.067855 23183 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.284s user 0.018s sys 0.031s
W20260812 19:13:11.090786 23290 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.760s user 0.008s sys 0.007s
W20260812 19:13:11.094911 23290 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.864s user 0.009s sys 0.008s
I20260812 19:13:11.103734 21112 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:11.099437 21204 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.120s user 0.010s sys 0.020s
I20260812 19:13:11.052446 23220 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:11.106148 23283 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.652s user 0.015s sys 0.015s
W20260812 19:13:11.108609 21204 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.238s user 0.012s sys 0.021s
W20260812 19:13:11.108994 23088 consensus_meta.cc:287] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.129s user 0.014s sys 0.006s
I20260812 19:13:11.110091 21150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54796 (request call id 1556) took 1938 ms. Trace:
W20260812 19:13:11.110937 21288 connection.cc:232] RPC call timeout handler was delayed by -1.5088989119976759s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17370000000000002s for a response
I20260812 19:13:11.111872 23258 raft_consensus.cc:3032] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:11.115017 21158 raft_consensus.cc:1226] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:11.115438 23255 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:11.115613 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:11.117139 23288 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:11.117086 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:11.118301 23254 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:11.118961 23257 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:11.120106 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:11.121459 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:11.099570 21287 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:11.125797 21153 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
W20260812 19:13:11.079912 23164 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.877s user 0.013s sys 0.017s
I20260812 19:13:11.127007 21150 rpcz_store.cc:276] 0812 19:13:09.172028 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:10.807339 (+1635311us) service_pool.cc:227] Handling call
0812 19:13:10.807920 (+ 581us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:11.061971 (+254051us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:11.109747 (+ 47776us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b540019d7d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:11.109972 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":582265728}
W20260812 19:13:11.127701 23283 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s user 0.016s sys 0.016s
I20260812 19:13:11.131282 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:11.134913 21323 raft_consensus.cc:1226] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:11.134716 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:11.135581 23287 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:11.123016 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:11.128098 21155 raft_consensus.cc:2454] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
W20260812 19:13:11.127225 23213 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.175s user 0.008s sys 0.025s
I20260812 19:13:11.154803 21320 raft_consensus.cc:2454] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:11.155011 23257 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:11.155720 21235 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1452) took 2053 ms (client timeout 2153 ms). Trace:
I20260812 19:13:11.132867 23258 raft_consensus.cc:729] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:11.138540 23255 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:11.158951 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 1.28867e+06us
I20260812 19:13:11.167703 21324 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:11.184556 21327 raft_consensus.cc:2379] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
W20260812 19:13:11.159083 23164 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.061s user 0.014s sys 0.018s
I20260812 19:13:11.184018 21318 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
W20260812 19:13:11.164119 21235 rpcz_store.cc:269] 0812 19:13:09.101893 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.304490 (+202597us) service_pool.cc:227] Handling call
0812 19:13:11.155207 (+1850717us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b54000b5c54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:11.155458 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4105841408}
W20260812 19:13:11.175760 23213 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.355s user 0.008s sys 0.026s
I20260812 19:13:11.191982 21169 tablet_copy_source_session.cc:205] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 19:13:11.192513 21322 raft_consensus.cc:3032] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:11.189308 21328 raft_consensus.cc:2454] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:11.175202 23201 raft_consensus.cc:2781] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:13:11.171698 23255 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:11.194918 21322 raft_consensus.cc:729] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:11.205271 21329 raft_consensus.cc:3037] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:11.209373 21155 raft_consensus.cc:1261] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:11.211211 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:11.212102 21150 raft_consensus.cc:3037] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:11.212471 21200 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:11.214833 23088 raft_consensus.cc:2781] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:11.214743 23209 consensus_queue.cc:1046] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:13:11.215507 23088 raft_consensus.cc:686] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:11.216257 23209 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:11.216450 23320 consensus_queue.cc:1035] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:13:11.215517 21149 raft_consensus.cc:1226] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:11.192602 21233 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
W20260812 19:13:11.223723 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 1.91465e+06us
I20260812 19:13:11.227970 21157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54796 (request call id 1561) took 1781 ms. Trace:
I20260812 19:13:11.228478 21157 rpcz_store.cc:276] 0812 19:13:09.446407 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:10.860772 (+1414365us) service_pool.cc:227] Handling call
0812 19:13:10.861241 (+ 469us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:11.066479 (+205238us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:11.227690 (+161211us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b54001248d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:11.227902 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":465134848}
W20260812 19:13:11.225653 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 94675.7us
W20260812 19:13:11.232757 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 1.17606e+06us
I20260812 19:13:11.235319 21158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54796 (request call id 1545) took 2273 ms. Trace:
I20260812 19:13:11.235899 21158 rpcz_store.cc:276] 0812 19:13:08.961703 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.341435 (+379732us) service_pool.cc:227] Handling call
0812 19:13:09.393096 (+ 51661us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:11.153414 (+1760318us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:11.234944 (+ 81530us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54001955d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:11.235211 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3867075968}
W20260812 19:13:11.237079 21233 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1355) took 2107 ms (client timeout 1832 ms). Trace:
W20260812 19:13:11.237584 21233 rpcz_store.cc:269] 0812 19:13:09.129939 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.145028 (+ 15089us) service_pool.cc:227] Handling call
0812 19:13:11.236781 (+2091753us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x7b54001509d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:11.236993 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4581342464}
I20260812 19:13:11.178539 23254 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:11.197160 23093 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:11.240645 22011 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:11.187031 23288 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:11.246397 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:13:11.257189 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:11.257488 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:11.216887 23088 consensus_queue.cc:237] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:11.230460 23201 raft_consensus.cc:686] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:11.262259 23316 raft_consensus.cc:3032] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Stepping down as leader of term 2
W20260812 19:13:11.265311 22008 connection.cc:232] RPC call timeout handler was delayed by -1.6025849230354652s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
W20260812 19:13:11.275799 22008 connection.cc:232] RPC call timeout handler was delayed by -1.5534565630368888s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1889s for a response
I20260812 19:13:11.268428 21149 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1645) took 1302 ms. Trace:
I20260812 19:13:11.262259 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:11.269326 21241 raft_consensus.cc:1226] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:11.272766 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:11.258881 21155 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:11.297864 21327 raft_consensus.cc:2396] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260812 19:13:11.216909 23209 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:11.230345 23287 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:11.276494 21149 rpcz_store.cc:276] 0812 19:13:09.966446 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:10.911059 (+944613us) service_pool.cc:227] Handling call
0812 19:13:10.911828 (+ 769us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:11.221581 (+309753us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:11.268138 (+ 46557us) spinlock_profiling.cc:243] Waited 303 ms on lock 0x7b54001941d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:11.268346 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":697759104}
W20260812 19:13:11.278293 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 221650us
I20260812 19:13:11.295409 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:11.279562 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:11.278389 23301 log.cc:924] Time spent T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.060s user 0.000s sys 0.001s
I20260812 19:13:11.305228 21319 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 3
W20260812 19:13:11.260360 23225 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.092s user 0.041s sys 0.087s
I20260812 19:13:11.260699 23199 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:11.284166 21328 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:11.306399 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 83166.5us
I20260812 19:13:11.275147 23024 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:11.308517 21324 raft_consensus.cc:2454] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:11.314832 23217 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:11.316767 21323 raft_consensus.cc:1261] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:11.317830 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:11.318988 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.348237 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:11.349080 21323 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:11.349458 23217 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:11.319331 21239 raft_consensus.cc:1226] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:11.350319 23143 consensus_queue.cc:1046] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:13:11.351826 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:11.351974 23217 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:11.351995 23297 raft_consensus.cc:3032] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:11.319588 21238 raft_consensus.cc:1261] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:11.319725 21236 raft_consensus.cc:1226] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:11.319950 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:11.320118 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:11.209436 23258 consensus_queue.cc:260] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:11.320441 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:11.320775 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:11.321328 23076 raft_consensus.cc:2781] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:11.321925 23254 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:11.323659 23220 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:11.325645 21150 raft_consensus.cc:2454] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:11.326431 23287 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:11.327706 21321 raft_consensus.cc:3032] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:11.372548 21321 raft_consensus.cc:729] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:11.372890 21169 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.BeginTabletCopySession from 127.0.0.1:51004 (request call id 185) took 2219 ms. Trace:
I20260812 19:13:11.373409 21169 rpcz_store.cc:276] 0812 19:13:09.153796 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.156749 (+ 2953us) service_pool.cc:227] Handling call
0812 19:13:11.372707 (+2215958us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:11.329077 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:11.330402 23257 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:11.378533 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:11.380702 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:11.381829 21320 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:11.332484 23209 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:11.333191 23236 consensus_meta.cc:287] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.548s user 0.013s sys 0.000s
I20260812 19:13:11.334501 21322 consensus_queue.cc:260] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:11.335455 21288 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:13:11.391306 21322 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:11.391382 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 56885.4us
I20260812 19:13:11.336011 21203 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:11.337241 23201 consensus_queue.cc:237] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:11.272766 23283 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:11.275444 23316 raft_consensus.cc:729] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:11.346275 21319 raft_consensus.cc:2454] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:11.346141 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:11.347685 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.352829 21239 raft_consensus.cc:2379] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 2: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:11.307042 23164 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:11.355110 21329 raft_consensus.cc:2454] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:11.359450 23288 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:11.359483 22045 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:13:11.359247 22011 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:11.362923 21235 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:13:11.363956 23309 consensus_queue.cc:1046] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:13:11.359382 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:11.411964 21238 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:11.362535 21287 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:11.360560 21243 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
I20260812 19:13:11.367053 23076 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:11.352653 23297 raft_consensus.cc:729] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:11.368798 23258 raft_consensus.cc:3037] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:11.370923 21112 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:11.278450 23082 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:11.339489 23225 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.303s user 0.044s sys 0.087s
I20260812 19:13:11.301380 23093 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:11.373951 21321 consensus_queue.cc:260] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:11.362498 23277 consensus_meta.cc:287] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.643s user 0.015s sys 0.000s
I20260812 19:13:11.350293 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:11.375607 21153 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:13:11.388949 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:11.409106 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:11.415469 22055 raft_consensus.cc:2379] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 4: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:11.415313 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:11.416064 21318 raft_consensus.cc:1261] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260812 19:13:11.425928 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 55204.8us
I20260812 19:13:11.433367 21321 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:11.426662 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:11.403146 23209 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:11.418046 21114 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:11.393440 23024 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:11.427454 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:11.415661 23076 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:11.441834 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 96319.4us
I20260812 19:13:11.413791 23217 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:11.442540 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:11.412386 23288 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:11.450399 21239 raft_consensus.cc:2379] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 2: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:11.449407 21329 raft_consensus.cc:1261] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:13:11.455205 21318 raft_consensus.cc:1226] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:11.427793 23287 raft_consensus.cc:2781] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:11.460067 21327 raft_consensus.cc:1226] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:11.461036 21323 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260812 19:13:11.462291 23160 raft_consensus.cc:2741] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260812 19:13:11.450460 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.462939 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:11.463645 22050 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:11.464946 21241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1350) took 3838 ms. Trace:
I20260812 19:13:11.465544 21241 rpcz_store.cc:276] 0812 19:13:07.627000 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:08.003493 (+376493us) service_pool.cc:227] Handling call
0812 19:13:08.630916 (+627423us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:11.305240 (+2674324us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:11.436020 (+130780us) spinlock_profiling.cc:243] Waited 2.64 s on lock 0x7b5400150ed4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:11.464838 (+ 28818us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6066621312}
I20260812 19:13:11.466971 21319 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 5: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:11.466750 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:11.472841 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:11.473995 22011 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:11.475276 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.475065 23288 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:11.476115 22046 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:11.419481 23283 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:11.479801 21240 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:13:11.481096 23283 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:11.481623 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 145761us
I20260812 19:13:11.482402 23217 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:11.484122 23176 raft_consensus.cc:2781] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:13:11.484711 23176 raft_consensus.cc:686] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:11.485725 23176 consensus_queue.cc:237] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:11.483456 21203 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:11.487053 23316 consensus_queue.cc:260] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:11.495795 21115 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:11.496039 23316 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:11.496778 23313 raft_consensus.cc:2781] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:11.497390 23313 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:11.497798 23313 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:11.489436 23093 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:11.490592 21154 raft_consensus.cc:1226] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:11.441748 23082 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:11.499831 21321 raft_consensus.cc:1261] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:13:11.504951 23082 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:11.511052 21237 raft_consensus.cc:2454] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:13:11.513085 23024 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:11.443449 22045 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40614 (request call id 367) took 2367 ms (client timeout 1737 ms). Trace:
W20260812 19:13:11.514370 22045 rpcz_store.cc:269] 0812 19:13:09.068793 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:09.077680 (+ 8887us) service_pool.cc:227] Handling call
0812 19:13:10.131013 (+1053333us) spinlock_profiling.cc:243] Waited 5.33 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:13:11.425381 (+1294368us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12269952}
I20260812 19:13:11.516583 23287 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:11.517124 23287 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:11.519706 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 108916us
I20260812 19:13:11.520144 23288 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:11.521324 23288 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:11.521651 21114 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:11.523238 21112 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:11.525029 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:11.525991 21243 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:11.526697 21235 raft_consensus.cc:1261] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:13:11.486402 23217 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:11.529723 21324 raft_consensus.cc:1226] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:11.532399 21169 tablet_copy_service.cc:342] P f741102bfe1441d1b453e5a838649059: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-4c656db261bd45f8b421339ba53dd87f received from {username='slave'} at 127.0.0.1:51004
I20260812 19:13:11.533005 21169 tablet_copy_service.cc:433] P f741102bfe1441d1b453e5a838649059: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-4c656db261bd45f8b421339ba53dd87f on tablet 4c656db261bd45f8b421339ba53dd87f with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:11.534430 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:11.535379 21158 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 3
I20260812 19:13:11.537348 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:11.543051 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.544108 22056 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:11.492251 23297 consensus_queue.cc:260] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:11.546561 23297 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:11.548442 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.549431 22052 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 5: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:11.492612 21328 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:11.492624 23208 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:11.493104 21203 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:11.551656 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:11.553406 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 60479.6us
I20260812 19:13:11.480969 23309 raft_consensus.cc:3032] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:11.567480 23024 raft_consensus.cc:2781] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:13:11.568502 23024 raft_consensus.cc:686] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:11.570118 23024 consensus_queue.cc:237] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:11.595654 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 99916.6us
I20260812 19:13:11.598035 21318 raft_consensus.cc:1226] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:11.478806 23307 consensus_queue.cc:1035] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:13:11.609809 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:11.429082 23164 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:11.619421 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:11.622804 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 101268us
I20260812 19:13:11.623761 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:11.626773 23329 log.cc:924] Time spent T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.125s user 0.000s sys 0.001s
W20260812 19:13:11.627151 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 192288us
I20260812 19:13:11.647657 21112 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:11.627890 23199 raft_consensus.cc:2781] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260812 19:13:11.648176 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 211342us
I20260812 19:13:11.629992 23288 raft_consensus.cc:2726] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:11.632213 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:11.649344 21203 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:11.565100 23309 raft_consensus.cc:729] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
W20260812 19:13:11.653364 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 177836us
W20260812 19:13:11.654515 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 57911.6us
I20260812 19:13:11.655592 21115 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:11.659752 21114 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:11.663832 21158 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:11.646211 21238 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
W20260812 19:13:11.666129 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 270002us
I20260812 19:13:11.666400 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:11.661993 21244 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:13:11.670938 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:11.671689 21237 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:11.616549 23225 raft_consensus.cc:2781] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:11.558255 23209 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:11.673502 21235 raft_consensus.cc:2379] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:11.652823 23199 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:11.675401 21153 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:13:11.678830 21115 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:11.683671 23217 raft_consensus.cc:2781] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:11.684266 23217 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:11.689260 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.674795 23225 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:11.691725 21288 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:13:11.689913 23160 raft_consensus.cc:2781] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:11.692755 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.693087 21112 leader_election.cc:302] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:11.679441 23196 raft_consensus.cc:2726] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:13:11.682687 23199 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:11.682224 23254 raft_consensus.cc:2781] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:11.692709 23160 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:11.703904 21154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54796 (request call id 1546) took 2737 ms. Trace:
I20260812 19:13:11.704272 23160 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:11.704389 21154 rpcz_store.cc:276] 0812 19:13:08.962057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:10.782318 (+1820261us) service_pool.cc:227] Handling call
0812 19:13:10.784623 (+ 2305us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:11.499705 (+715082us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:11.694662 (+194957us) spinlock_profiling.cc:243] Waited 706 ms on lock 0x7b54003cf854. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:11.699088 (+ 4426us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1622732544}
I20260812 19:13:11.693624 22051 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:11.675891 23088 consensus_queue.cc:1046] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260812 19:13:11.706204 22047 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:11.686411 23288 raft_consensus.cc:2726] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260812 19:13:11.713793 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 140949us
I20260812 19:13:11.688802 23213 raft_consensus.cc:2781] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:11.690514 23217 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:11.695169 23225 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:11.719427 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.726320 21115 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:11.726969 22053 raft_consensus.cc:2454] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:13:11.719650 23220 raft_consensus.cc:2781] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:11.727921 23220 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:11.728385 23220 raft_consensus.cc:3037] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:11.728513 23287 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:11.729206 23209 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:11.729874 23209 consensus_queue.cc:1035] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:13:11.731187 21203 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:11.715858 23304 raft_consensus.cc:2781] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:11.722007 23208 raft_consensus.cc:2726] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:11.716202 23335 raft_consensus.cc:2726] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:11.722787 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.743654 22051 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:11.743202 23309 consensus_queue.cc:260] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:11.723636 23076 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:11.744685 23309 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:11.720799 23164 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:11.732017 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 369604us
W20260812 19:13:11.747437 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 54455us
I20260812 19:13:11.747509 21287 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:11.733462 23334 raft_consensus.cc:2781] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:13:11.749279 23334 raft_consensus.cc:686] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:11.749804 23088 raft_consensus.cc:2726] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:13:11.750571 23334 consensus_queue.cc:237] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:11.718176 23213 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:11.716226 23255 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
W20260812 19:13:11.734301 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 74703.6us
I20260812 19:13:11.735471 23313 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:11.739358 22047 raft_consensus.cc:1261] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:11.742413 23303 raft_consensus.cc:2781] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:11.762094 21112 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:11.768602 22053 raft_consensus.cc:1226] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:11.746279 23076 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:11.778733 23243 raft_consensus.cc:2781] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:11.778914 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.779202 23243 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:11.779562 23243 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:11.780820 23287 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:11.782152 21203 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:11.783283 23196 raft_consensus.cc:2781] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:11.783699 23196 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:11.784065 23196 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:11.787498 21112 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:11.768895 23213 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:11.794590 21114 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:11.799335 23160 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:11.799561 23217 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:11.774461 23335 raft_consensus.cc:2726] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:11.803056 22048 raft_consensus.cc:2379] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 2: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:11.802937 23217 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:11.825438 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.826937 23313 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:11.803160 23320 raft_consensus.cc:3032] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
W20260812 19:13:11.831133 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 43744us
I20260812 19:13:11.832098 21112 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:11.838403 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 148760us
I20260812 19:13:11.839167 21112 leader_election.cc:302] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:11.835933 23342 log.cc:924] Time spent T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20260812 19:13:11.841089 21239 raft_consensus.cc:1261] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:13:11.842233 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.843143 23217 raft_consensus.cc:2781] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:11.846226 22050 raft_consensus.cc:1261] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:11.847054 23254 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:11.830570 23320 raft_consensus.cc:729] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:11.850996 21288 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
W20260812 19:13:11.859032 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 77060.9us
I20260812 19:13:11.860908 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.861686 22051 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:11.862993 23225 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:11.870950 23255 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:11.871300 23243 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:11.864223 23160 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:11.872484 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:11.873046 23160 raft_consensus.cc:2781] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:11.873199 22050 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:11.873601 23160 raft_consensus.cc:686] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:11.873679 23243 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:11.874838 23160 consensus_queue.cc:237] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:11.875614 23347 raft_consensus.cc:2726] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:11.876415 23347 consensus_queue.cc:1035] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:13:11.874401 23243 raft_consensus.cc:2781] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:13:11.878104 23243 raft_consensus.cc:686] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:11.879199 23243 consensus_queue.cc:237] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:11.870045 23220 raft_consensus.cc:504] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:11.852627 23217 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:11.859476 23254 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:11.858456 23303 raft_consensus.cc:686] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:13:11.896130 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 56858.1us
W20260812 19:13:11.911934 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 121300us
I20260812 19:13:11.913053 21114 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:11.906661 23335 raft_consensus.cc:2781] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:11.913869 23335 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:11.914297 23335 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:11.913983 23196 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:11.914953 21203 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:11.914711 23303 consensus_queue.cc:237] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:11.916023 21203 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:11.916775 23196 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:11.917470 23196 raft_consensus.cc:2781] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:11.917517 21203 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:11.917965 23196 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:11.887097 23225 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:11.912802 23199 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:11.920051 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 50909.4us
I20260812 19:13:11.904999 23349 consensus_queue.cc:1035] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260812 19:13:11.903064 23304 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:11.924629 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:11.907480 23217 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:11.918445 23196 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:11.933342 21114 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:11.942670 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 84709.3us
I20260812 19:13:11.928109 23304 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:11.953372 21200 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:11.954525 23024 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:13:11.954957 23274 raft_consensus.cc:2781] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:11.955477 23274 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:11.955866 23274 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:11.956506 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.957361 22052 raft_consensus.cc:2454] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:11.958153 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.946511 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.947817 23257 raft_consensus.cc:2781] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:11.947870 23352 raft_consensus.cc:2726] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:11.949483 23199 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:11.941179 23269 raft_consensus.cc:2781] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:11.950453 21114 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:11.974504 21319 raft_consensus.cc:1261] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:11.951331 22011 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:11.951982 23220 leader_election.cc:288] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:11.952359 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:11.975390 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.953583 21115 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:11.946883 22056 raft_consensus.cc:1261] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:11.950351 21287 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:11.958382 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:11.961735 23213 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:11.963261 21112 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:11.963322 23255 consensus_queue.cc:1046] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:13:11.966552 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:11.967430 22055 raft_consensus.cc:1261] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:11.985538 21327 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:11.967933 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:11.968633 21318 raft_consensus.cc:1261] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:13:11.969583 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:11.970175 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:11.970811 21320 raft_consensus.cc:1261] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:13:11.971340 21321 raft_consensus.cc:1261] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260812 19:13:11.971873 21324 raft_consensus.cc:1261] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:11.972046 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:11.976614 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:11.946394 23225 raft_consensus.cc:2726] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 19:13:11.980225 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 62825.1us
I20260812 19:13:11.961723 23347 consensus_queue.cc:1035] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260812 19:13:11.979753 23257 raft_consensus.cc:686] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:12.004647 21323 raft_consensus.cc:2379] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 2: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:11.976706 23269 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:12.004994 23257 consensus_queue.cc:237] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:12.005934 23269 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:11.981842 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:12.007201 21319 raft_consensus.cc:2396] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260812 19:13:11.982100 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:12.008363 22056 raft_consensus.cc:2379] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 4: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:11.982950 22053 raft_consensus.cc:2379] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 2: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:11.983593 23213 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:11.984308 23199 raft_consensus.cc:2726] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260812 19:13:11.984443 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 45172.1us
I20260812 19:13:12.010262 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:12.011090 21114 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:12.011781 23335 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:12.013763 21114 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:11.986383 21329 raft_consensus.cc:2454] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:11.990839 23309 raft_consensus.cc:2781] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:12.015102 23309 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:12.015529 23309 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:12.015851 23024 consensus_queue.cc:1046] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:13:11.990870 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:11.992133 23352 raft_consensus.cc:3032] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:12.017081 23352 raft_consensus.cc:729] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:12.017300 21327 raft_consensus.cc:2379] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:12.017416 23024 raft_consensus.cc:3032] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:12.017903 23024 raft_consensus.cc:729] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:12.018332 23213 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:12.018126 23352 consensus_queue.cc:260] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:12.005673 21033 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48434 (request call id 250) took 3230 ms. Trace:
I20260812 19:13:12.019310 23225 consensus_queue.cc:1046] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:13:11.994771 22047 raft_consensus.cc:3037] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:12.020576 23225 raft_consensus.cc:2741] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260812 19:13:12.021281 23225 raft_consensus.cc:3032] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:12.021667 23225 raft_consensus.cc:729] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:11.994827 23349 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:12.022656 23225 consensus_queue.cc:260] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:12.019636 23352 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:12.024686 23225 raft_consensus.cc:3037] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:11.996333 21112 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:12.026330 23347 raft_consensus.cc:2781] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:12.026499 21320 raft_consensus.cc:1226] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260812 19:13:12.026937 23347 raft_consensus.cc:686] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:12.028046 23347 consensus_queue.cc:237] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:11.958873 23320 consensus_queue.cc:260] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:12.031478 23320 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:12.032184 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:12.032998 21327 raft_consensus.cc:2454] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:12.034132 23316 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:12.019351 21033 rpcz_store.cc:276] 0812 19:13:08.775146 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:08.775354 (+ 208us) service_pool.cc:227] Handling call
0812 19:13:12.005408 (+3230054us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0812 19:13:10.035932 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0812 19:13:10.195357 (+159425us) write_op.cc:432] Acquiring schema lock in shared mode
0812 19:13:10.195417 (+ 60us) write_op.cc:435] Acquired schema lock
0812 19:13:10.195437 (+ 20us) tablet.cc:660] Decoding operations
0812 19:13:10.196260 (+ 823us) write_op.cc:620] Acquiring the partition lock for write op
0812 19:13:10.196388 (+ 128us) write_op.cc:641] Partition lock acquired for write op
0812 19:13:10.196450 (+ 62us) tablet.cc:683] Acquiring locks for 1 operations
0812 19:13:10.196806 (+ 356us) tablet.cc:699] Row locks acquired
0812 19:13:10.196828 (+ 22us) write_op.cc:260] PREPARE: finished
0812 19:13:10.197052 (+ 224us) write_op.cc:270] Start()
0812 19:13:10.197294 (+ 242us) write_op.cc:276] Timestamp: P: 1786561990196984 usec, L: 0
0812 19:13:10.197321 (+ 27us) op_driver.cc:331] REPLICATION: starting
0812 19:13:10.198393 (+ 1072us) log.cc:842] Serialized 599 byte log entry
0812 19:13:11.707398 (+1509005us) op_driver.cc:444] REPLICATION: finished
0812 19:13:11.908404 (+201006us) write_op.cc:301] APPLY: starting
0812 19:13:11.918766 (+ 10362us) tablet.cc:1365] starting BulkCheckPresence
0812 19:13:11.918908 (+ 142us) tablet.cc:1368] finished BulkCheckPresence
0812 19:13:11.919065 (+ 157us) tablet.cc:1370] starting ApplyRowOperation cycle
0812 19:13:11.933178 (+ 14113us) tablet.cc:1381] finished ApplyRowOperation cycle
0812 19:13:11.933463 (+ 285us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0812 19:13:11.933510 (+ 47us) write_op.cc:312] APPLY: finished
0812 19:13:11.933918 (+ 408us) log.cc:842] Serialized 23 byte log entry
0812 19:13:11.934091 (+ 173us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:13:11.934238 (+ 147us) write_op.cc:454] Released schema lock
0812 19:13:11.934311 (+ 73us) write_op.cc:341] FINISH: Updating metrics
0812 19:13:11.934808 (+ 497us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:13:11.934850 (+ 42us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":152696,"apply.run_cpu_time_us":3233,"apply.run_wall_time_us":48336,"num_ops":1,"prepare.queue_time_us":878409,"prepare.run_cpu_time_us":23246,"prepare.run_wall_time_us":1065567,"raft.queue_time_us":210837,"raft.run_cpu_time_us":23442,"raft.run_wall_time_us":116328,"replication_time_us":1509922,"thread_start_us":1618869,"threads_started":4,"wal-append.queue_time_us":1163734}]]}
I20260812 19:13:12.036047 23209 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:12.036675 23209 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:12.033680 23024 consensus_queue.cc:260] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:12.037964 23283 consensus_queue.cc:1046] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:13:12.039721 23173 consensus_queue.cc:1046] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:12.041239 23316 raft_consensus.cc:3032] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:12.042550 23316 raft_consensus.cc:729] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:12.044942 23316 consensus_queue.cc:260] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:12.046120 23316 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:11.997143 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 43657.8us
I20260812 19:13:11.999183 23164 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:12.053733 23164 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:12.054343 23164 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:12.054417 23088 consensus_queue.cc:1035] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 19:13:12.055492 21329 raft_consensus.cc:1261] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260812 19:13:12.055960 21329 pending_rounds.cc:76] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412: Aborting all ops after (but not including) 1
I20260812 19:13:12.056258 21329 pending_rounds.cc:97] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260812 19:13:12.056305 23024 raft_consensus.cc:3037] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:12.056675 21329 raft_consensus.cc:2866] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260812 19:13:11.999413 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 48253.8us
I20260812 19:13:12.058223 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:12.058928 21321 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:12.059172 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:12.058872 23304 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:12.059504 22011 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:12.059811 21328 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:12.060464 22011 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:12.061369 23304 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:12.062347 23014 tablet_copy_client.cc:531] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:13:12.062974 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:11.975757 22046 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:12.009224 23254 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:12.018797 23213 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:11.993604 23255 raft_consensus.cc:2726] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:13:11.995025 23220 consensus_queue.cc:1035] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:13:11.995821 21287 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:12.037768 21114 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:12.064304 21200 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:12.105509 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:12.096320 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:12.106355 21318 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:12.106823 21327 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:12.104769 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:12.106158 21328 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:12.108807 21324 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:12.109802 21321 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:12.104336 23313 raft_consensus.cc:2781] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:12.118975 23313 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:12.119542 23313 raft_consensus.cc:3037] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:12.055518 23335 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:12.122592 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:12.121053 21319 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:12.127360 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:12.127887 21153 raft_consensus.cc:2379] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:12.128301 21322 raft_consensus.cc:3037] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:12.124235 21112 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:12.132629 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:12.133631 22053 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:12.098748 23213 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
W20260812 19:13:12.142249 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 104570us
I20260812 19:13:12.146374 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:12.147507 21158 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:12.165939 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:12.166923 21150 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
W20260812 19:13:12.167943 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 172272us
I20260812 19:13:12.122143 23088 raft_consensus.cc:2781] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:12.112363 22047 raft_consensus.cc:2454] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:12.175949 21112 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:12.181483 22051 raft_consensus.cc:1226] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:12.179052 21112 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:12.140272 23334 raft_consensus.cc:2726] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:12.141510 23209 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:12.154048 23160 raft_consensus.cc:2781] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:12.152798 23255 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:12.191432 21112 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:12.128842 23283 consensus_queue.cc:1035] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260812 19:13:12.128082 23173 consensus_queue.cc:1035] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260812 19:13:12.210997 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:12.187538 23303 raft_consensus.cc:2781] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260812 19:13:12.194409 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 46497.1us
I20260812 19:13:12.199651 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:12.212755 21153 raft_consensus.cc:2454] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:12.171347 23088 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:12.165130 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:12.203177 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:12.215358 21150 raft_consensus.cc:2379] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:12.169169 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:12.195693 23288 raft_consensus.cc:2781] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:12.204458 21114 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:12.185194 23307 raft_consensus.cc:2741] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260812 19:13:12.217515 23088 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:12.197680 23160 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:12.141808 23309 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:12.163033 23243 consensus_queue.cc:1035] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260812 19:13:12.185503 23304 consensus_queue.cc:1035] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
W20260812 19:13:12.270998 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 79714.4us
I20260812 19:13:12.283193 21112 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:12.273543 23348 raft_consensus.cc:2726] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:12.209127 23209 consensus_queue.cc:1035] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.074s
I20260812 19:13:12.272091 23220 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:12.273921 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:12.214986 23257 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:12.272769 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 68446.1us
I20260812 19:13:12.214563 23303 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:12.277591 21322 raft_consensus.cc:2454] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:12.277807 21155 raft_consensus.cc:1226] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:12.279925 21328 raft_consensus.cc:1226] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:12.297295 21319 raft_consensus.cc:1226] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:12.230491 22011 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:12.284345 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:12.286293 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 155773us
I20260812 19:13:12.231963 22048 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:12.303491 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 90615.2us
I20260812 19:13:12.304687 22011 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:12.205472 23335 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:12.274112 23199 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:12.306074 23335 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:12.306135 23199 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:12.309696 23199 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:12.313064 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:12.275792 21323 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:12.281756 23351 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:12.279243 23160 raft_consensus.cc:3037] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:12.311587 23199 raft_consensus.cc:2781] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:12.285472 23288 raft_consensus.cc:686] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:12.276465 23350 raft_consensus.cc:2781] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:12.342139 21328 raft_consensus.cc:1226] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 4. Ops: []
I20260812 19:13:12.325887 23220 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:12.313701 23303 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:12.362531 23208 consensus_queue.cc:1035] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260812 19:13:12.368846 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:12.369488 21238 raft_consensus.cc:1261] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260812 19:13:12.369807 21239 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:12.370829 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:12.371868 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:12.372515 21236 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 4
I20260812 19:13:12.367808 21158 raft_consensus.cc:1226] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:12.341144 23257 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:12.388947 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:12.389940 21244 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:12.390590 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:12.389307 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:12.393628 21235 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:12.377213 23257 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:12.391352 21243 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:12.344882 23254 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:12.392009 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:12.312363 22048 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 4.
I20260812 19:13:12.345917 23199 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:12.408072 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:12.351300 23220 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:12.398041 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:12.405545 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:12.412883 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 108298us
I20260812 19:13:12.437366 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:12.438405 21239 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
W20260812 19:13:12.441680 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 142964us
I20260812 19:13:12.442745 21155 raft_consensus.cc:2354] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4.
I20260812 19:13:12.444406 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:12.390043 23350 raft_consensus.cc:686] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:12.451344 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:12.346591 23217 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:12.447649 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:12.453758 21288 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:13:12.414165 23199 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:12.387509 23352 consensus_queue.cc:1035] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260812 19:13:12.351262 23288 consensus_queue.cc:237] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:12.343909 23351 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:12.457800 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:12.463483 22050 consensus_meta.cc:287] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.590s user 0.014s sys 0.000s
I20260812 19:13:12.458540 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:12.482122 21328 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 5: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:12.475533 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:12.477504 21236 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:12.477557 22053 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:12.477962 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:12.484454 21239 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:12.471028 22011 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:12.480063 21200 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:12.480098 21154 raft_consensus.cc:1226] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:12.491735 21233 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:12.497551 21200 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:12.498075 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 61425.2us
I20260812 19:13:12.498879 21200 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:12.499657 22011 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:12.484287 23258 consensus_queue.cc:1046] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260812 19:13:12.508023 23217 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:12.508498 23196 consensus_meta.cc:287] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.549s user 0.010s sys 0.001s
I20260812 19:13:12.508936 23196 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:12.510864 23255 raft_consensus.cc:2726] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:12.485087 22050 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:12.458878 23287 raft_consensus.cc:2781] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:12.448733 23164 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:12.500087 21155 raft_consensus.cc:1226] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:12.480089 23304 raft_consensus.cc:2781] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:12.499351 21151 raft_consensus.cc:2454] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:12.502506 23258 raft_consensus.cc:3032] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:12.569200 21115 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:12.569969 23350 consensus_queue.cc:237] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:12.579005 21238 raft_consensus.cc:2379] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:12.579110 21318 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:12.630362 21150 raft_consensus.cc:1226] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:12.580615 21239 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:12.509984 23088 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:12.576818 21324 raft_consensus.cc:2454] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:12.577217 21244 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:12.631567 21233 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
W20260812 19:13:12.561757 23349 consensus_meta.cc:287] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.539s user 0.006s sys 0.006s
W20260812 19:13:12.636777 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 137975us
I20260812 19:13:12.594880 23369 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:12.595851 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:12.596920 23014 tablet_bootstrap.cc:492] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:13:12.597087 23196 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:12.634981 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 65844.9us
I20260812 19:13:12.603201 23258 raft_consensus.cc:729] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:12.639040 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:12.653753 21237 raft_consensus.cc:1226] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:12.639539 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:12.602834 23304 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:12.643206 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:12.586020 23309 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:12.653311 21115 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:12.667956 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:12.669484 21320 raft_consensus.cc:1226] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: [3.4-3.4]
W20260812 19:13:12.670563 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 171263us
I20260812 19:13:12.647029 21114 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:12.648656 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:12.563309 23209 raft_consensus.cc:2781] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:12.581600 23283 raft_consensus.cc:2726] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:12.650442 22047 raft_consensus.cc:1261] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:13:12.602762 23370 raft_consensus.cc:2726] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:12.671134 23352 raft_consensus.cc:2781] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:12.673640 23196 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:12.674106 21203 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:12.674654 23363 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:12.674180 23258 consensus_queue.cc:260] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:12.646906 23349 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:12.658768 21152 raft_consensus.cc:1261] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:13:12.711730 21318 raft_consensus.cc:2454] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
I20260812 19:13:12.676802 21328 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:12.716504 21114 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:12.649914 23255 consensus_queue.cc:1035] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260812 19:13:12.646485 23335 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:12.689911 23368 raft_consensus.cc:2741] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20260812 19:13:12.690552 23360 log.cc:924] Time spent T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.223s user 0.001s sys 0.000s
I20260812 19:13:12.690096 23196 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:12.691406 23367 log.cc:924] Time spent T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
W20260812 19:13:12.730705 23213 consensus_meta.cc:287] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.543s user 0.012s sys 0.000s
I20260812 19:13:12.691978 23173 consensus_queue.cc:1046] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:12.692028 21156 raft_consensus.cc:1261] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:13:12.695750 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:12.697750 23164 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:12.701721 23356 log.cc:924] Time spent T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.550s user 0.003s sys 0.000s
I20260812 19:13:12.660684 23304 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:12.722334 21200 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:12.675355 23209 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:12.712361 23287 raft_consensus.cc:686] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 4 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
W20260812 19:13:12.724946 23274 consensus_meta.cc:287] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.769s user 0.011s sys 0.002s
I20260812 19:13:12.725013 21203 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:12.660756 23024 consensus_queue.cc:1046] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260812 19:13:12.731890 21328 raft_consensus.cc:1261] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260812 19:13:12.684496 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 401359us
I20260812 19:13:12.724541 23375 raft_consensus.cc:2781] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:12.743885 21324 raft_consensus.cc:2454] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
W20260812 19:13:12.691792 23361 log.cc:924] Time spent T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.227s user 0.001s sys 0.000s
I20260812 19:13:12.687356 23372 raft_consensus.cc:2781] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260812 19:13:12.724181 23316 consensus_meta.cc:287] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.678s user 0.014s sys 0.003s
W20260812 19:13:12.725185 23225 consensus_meta.cc:287] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.700s user 0.010s sys 0.005s
I20260812 19:13:12.717948 23352 raft_consensus.cc:686] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:12.652575 23369 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:12.762341 21327 raft_consensus.cc:2454] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:13:12.729949 21321 raft_consensus.cc:1226] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260812 19:13:12.775019 23366 raft_consensus.cc:2781] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:12.775722 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:12.776695 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 104528us
W20260812 19:13:12.777549 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 53831us
I20260812 19:13:12.781787 23370 raft_consensus.cc:2781] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:12.728405 23309 consensus_queue.cc:1035] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260812 19:13:12.761729 23258 raft_consensus.cc:3037] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:12.759853 23213 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:12.764139 23209 raft_consensus.cc:3037] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:12.780416 23375 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:12.714041 23363 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:12.783007 23176 raft_consensus.cc:3032] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:12.778100 23351 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:12.795918 23217 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:12.813741 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:12.796886 21320 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:12.786911 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:12.785775 23372 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:12.819470 21203 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:12.830259 21323 raft_consensus.cc:1226] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:12.821722 21114 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:12.831818 23364 raft_consensus.cc:2781] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:12.832404 23364 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:12.832330 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:12.832916 23364 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:12.827193 23375 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:12.833586 23369 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:12.833863 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:12.835393 23373 log.cc:924] Time spent T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
I20260812 19:13:12.827546 23176 raft_consensus.cc:729] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:12.835254 23352 consensus_queue.cc:237] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:12.836632 23176 consensus_queue.cc:260] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:12.838115 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:12.839305 21328 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:12.840461 21321 raft_consensus.cc:1261] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:13:12.842137 21321 raft_consensus.cc:1261] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:13:12.842201 21239 raft_consensus.cc:1261] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:13:12.842881 23088 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:12.844357 21235 raft_consensus.cc:1261] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:13:12.844990 23372 raft_consensus.cc:3037] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:12.791667 23283 raft_consensus.cc:2781] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:12.846899 23283 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:12.847524 23283 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:12.845033 23217 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:12.847783 23287 consensus_queue.cc:237] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:12.825415 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:12.851190 23379 log.cc:924] Time spent T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260812 19:13:12.852763 21155 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:12.811601 23380 raft_consensus.cc:2781] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:12.816926 23366 raft_consensus.cc:686] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:12.854794 23366 consensus_queue.cc:237] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:12.813052 23376 raft_consensus.cc:2781] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:12.834509 21320 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:12.813453 23225 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:12.810735 23370 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:12.860342 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:12.874485 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 52875.7us
I20260812 19:13:12.845981 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:12.831046 22011 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:12.794957 21112 leader_election.cc:302] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:12.838207 23176 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:12.860373 23225 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:12.888037 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 93178.1us
I20260812 19:13:12.886274 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:12.886874 23255 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:12.776593 23274 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:12.885820 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:12.894348 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:12.861903 23335 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:12.899253 23160 consensus_meta.cc:287] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.520s user 0.013s sys 0.000s
I20260812 19:13:12.887667 23213 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:12.902863 23217 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:12.887948 23349 raft_consensus.cc:2781] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:12.903805 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:12.904457 23255 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:12.859030 23376 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:12.895423 21320 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 5: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:12.908710 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:12.907505 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 96436.9us
I20260812 19:13:12.909224 21112 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:12.921322 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 146819us
I20260812 19:13:12.880682 23380 raft_consensus.cc:686] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:12.950990 21328 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:12.906883 23196 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:13.046064 21244 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:13.263970 21329 raft_consensus.cc:2379] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 4: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:12.883795 23370 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:12.982517 21204 leader_election.cc:334] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Timed out: RequestConsensusVote RPC to 127.19.70.132:39579 timed out after 1.622s (SENT)
I20260812 19:13:13.315428 21204 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
W20260812 19:13:13.030869 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 212847us
I20260812 19:13:13.265862 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:12.978796 23320 consensus_meta.cc:287] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.947s user 0.017s sys 0.000s
I20260812 19:13:12.908097 22047 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:13.274355 21318 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.015s sys 0.012s
I20260812 19:13:13.276381 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:13:13.341382 21318 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s user 0.016s sys 0.012s
I20260812 19:13:13.341653 21156 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:13.071516 23382 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:12.926775 23390 consensus_queue.cc:1035] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.367s
I20260812 19:13:12.978721 23376 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:13.279199 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:13.279848 23380 consensus_queue.cc:237] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:13.281476 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 372382us
I20260812 19:13:12.990516 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:13.069454 23269 consensus_meta.cc:287] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.063s user 0.010s sys 0.014s
I20260812 19:13:12.911181 23351 consensus_queue.cc:1035] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.369s
I20260812 19:13:13.286303 23220 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:13.282810 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:13.339385 23313 consensus_meta.cc:287] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.202s user 0.011s sys 0.007s
I20260812 19:13:13.283797 21150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1662) took 1063 ms. Trace:
I20260812 19:13:13.261993 23381 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:13.295171 23400 consensus_queue.cc:1046] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:13:13.296566 23257 consensus_queue.cc:1035] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:13:13.298843 23208 consensus_queue.cc:1035] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:13.300784 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50042 (request call id 201) took 1092 ms. Trace:
I20260812 19:13:13.302233 21151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1665) took 1017 ms. Trace:
W20260812 19:13:13.304212 22011 connection.cc:232] RPC call timeout handler was delayed by -2.1207735750358552s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20790000000000003s for a response
I20260812 19:13:13.282585 23196 consensus_queue.cc:1035] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260812 19:13:12.911638 23368 raft_consensus.cc:2741] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260812 19:13:12.959524 22008 leader_election.cc:334] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.695s (SENT)
I20260812 19:13:12.895843 23385 raft_consensus.cc:2781] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:13.346916 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:13.357822 21112 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:13.362355 21150 rpcz_store.cc:276] 0812 19:13:12.220190 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.225519 (+ 5329us) service_pool.cc:227] Handling call
0812 19:13:12.281130 (+ 55611us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:12.630932 (+349802us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:13.283528 (+652596us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b5400326254. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:13.283724 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":802628352}
I20260812 19:13:13.289420 21033 catalog_manager.cc:5495] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 2 to 3, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 3 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:13:13.363145 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:13.157335 23346 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:13.366025 21328 raft_consensus.cc:1261] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:13:12.922531 23335 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:13.007949 23349 raft_consensus.cc:686] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:12.926468 23160 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:13.244016 23364 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:13.419705 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:13:13.619252 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 261545us
I20260812 19:13:13.619580 21154 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:13.620677 21323 raft_consensus.cc:2379] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 4: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:13.546742 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:13.566838 21149 consensus_meta.cc:287] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.773s user 0.004s sys 0.009s
W20260812 19:13:13.616080 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 633582us
W20260812 19:13:13.623816 22011 connection.cc:232] RPC call timeout handler was delayed by -2.0433449180563912s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2073s for a response
I20260812 19:13:13.338004 23088 raft_consensus.cc:2726] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:13.584342 21238 raft_consensus.cc:1226] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 4. Ops: []
I20260812 19:13:13.618392 21151 rpcz_store.cc:276] 0812 19:13:12.284412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.284565 (+ 153us) service_pool.cc:227] Handling call
0812 19:13:13.302023 (+1017458us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b5400214b54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:13.302154 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":426476416}
I20260812 19:13:13.459213 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:13.656193 21327 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:13.622774 21149 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:13.366899 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:13.621986 21155 raft_consensus.cc:2379] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 2: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:13.657699 21233 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:13.659097 22056 raft_consensus.cc:1261] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260812 19:13:13.659723 21112 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:13.641122 21156 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:13.357661 21320 raft_consensus.cc:2362] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3. Re-sending same reply.
I20260812 19:13:13.649683 21158 raft_consensus.cc:1226] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:13.651363 21033 catalog_manager.cc:5495] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 1 to 2, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 2 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:13:13.652480 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:13.662237 21328 raft_consensus.cc:2379] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 4: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:13.662079 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:13.663110 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:13.653035 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:13.663182 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:13.359078 23225 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:13.664762 21033 catalog_manager.cc:5495] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 2 to 3, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 3 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:13:13.667544 21323 raft_consensus.cc:2454] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:13:13.670851 21033 catalog_manager.cc:5495] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 1 to 2, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 2 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:13:13.668273 21112 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:13.648573 21319 rpcz_store.cc:276] 0812 19:13:12.213955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.217957 (+ 4002us) service_pool.cc:227] Handling call
0812 19:13:12.297042 (+ 79085us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:12.297825 (+ 783us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:13.300486 (+1002661us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b540023f5d0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:13.300671 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":49975168}
I20260812 19:13:13.699592 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260812 19:13:13.718248 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 758731us
W20260812 19:13:13.625662 23258 consensus_meta.cc:287] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.810s user 0.011s sys 0.004s
W20260812 19:13:13.719147 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 54612.5us
I20260812 19:13:13.720126 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:13.720502 21204 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:13.720819 21323 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:13.721557 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 54824.2us
I20260812 19:13:13.362076 23320 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:13.609038 23391 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:13.722426 21203 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:13.725344 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:13.725812 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:13.542881 23255 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:13.726135 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:13.659122 22011 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.70.129:36551 timed out after 1.970s (SENT) [suppressed 20 similar messages]
I20260812 19:13:13.480296 23220 raft_consensus.cc:686] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:13.731637 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:13.616981 23334 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:13.458704 23363 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:13.421730 23313 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:13.364953 23269 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:13.731963 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:13.850023 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:13.350879 23382 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:13.843063 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:13.777174 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:13.894234 21158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1670) took 1409 ms. Trace:
I20260812 19:13:13.894768 21327 raft_consensus.cc:3037] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 4
I20260812 19:13:13.855780 21154 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:13.851495 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:13.733412 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:13.896397 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:13.875587 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:13.855507 21233 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 4.
I20260812 19:13:13.856897 21151 raft_consensus.cc:2379] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:13.732905 21321 raft_consensus.cc:2454] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
W20260812 19:13:13.876979 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 208780us
I20260812 19:13:13.658671 23385 raft_consensus.cc:686] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:13.650893 23274 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:13.815122 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:13.886485 21288 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.013s sys 0.041s
I20260812 19:13:13.652169 23390 raft_consensus.cc:2781] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:13.727572 23014 tablet_bootstrap.cc:700] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: Time spent opening tablet: real 0.426s user 0.002s sys 0.003s
I20260812 19:13:13.723213 23350 raft_consensus.cc:2726] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:13.737588 21328 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:13.894641 21158 rpcz_store.cc:276] 0812 19:13:12.484589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.484750 (+ 161us) service_pool.cc:227] Handling call
0812 19:13:12.488041 (+ 3291us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:13.664246 (+1176205us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:13.893984 (+229738us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54002cc754. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:13.894149 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2670953344}
W20260812 19:13:13.878526 23199 consensus_meta.cc:287] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.392s user 0.018s sys 0.006s
W20260812 19:13:13.912580 21288 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.682s user 0.013s sys 0.041s
I20260812 19:13:13.912472 21319 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:13:13.777580 23401 raft_consensus.cc:2726] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:13.915000 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:13.939690 21154 raft_consensus.cc:2396] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260812 19:13:13.847030 23392 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:13.805315 23213 raft_consensus.cc:2781] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:13.778623 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:13.911428 23287 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.902s user 0.010s sys 0.002s
I20260812 19:13:13.922775 21323 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:13.925063 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 220) took 1024 ms. Trace:
I20260812 19:13:13.913688 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1657) took 1203 ms. Trace:
I20260812 19:13:13.954809 21319 rpcz_store.cc:276] 0812 19:13:12.900950 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:13.737310 (+836360us) service_pool.cc:227] Handling call
0812 19:13:13.924977 (+187667us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:13.920102 22008 leader_election.cc:334] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.727s (SENT)
W20260812 19:13:13.940371 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 218013us
I20260812 19:13:13.940526 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:13.914206 23304 consensus_meta.cc:287] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.131s user 0.010s sys 0.008s
I20260812 19:13:13.896412 23208 raft_consensus.cc:2726] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:13.893628 23196 raft_consensus.cc:2726] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:13.920034 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:13.952865 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:13.953272 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:13.952539 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:13.954881 21320 rpcz_store.cc:276] 0812 19:13:12.710426 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:13.912606 (+1202180us) service_pool.cc:227] Handling call
0812 19:13:13.912982 (+ 376us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:13.913243 (+ 261us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:13.913282 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:13.913338 (+ 56us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:13.913394 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:13.913467 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:13.913611 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:13.775774 23320 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:13.658548 23346 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:13.885486 23334 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:13.651487 23381 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:13.780994 23391 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:13.923903 23349 consensus_queue.cc:237] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:13.938335 22050 raft_consensus.cc:1226] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 1. Current term is 2. Ops: []
W20260812 19:13:14.014139 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 60100.3us
I20260812 19:13:14.342881 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:14.374897 21150 raft_consensus.cc:2396] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260812 19:13:14.025853 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:14.386027 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:14.386559 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:14.387373 21154 raft_consensus.cc:2379] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 4: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:14.388176 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:14.388798 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:14.389284 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:14.389868 21154 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:14.390015 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:14.289956 21237 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:13:14.391036 21237 raft_consensus.cc:595] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:13:14.391311 21235 raft_consensus.cc:1226] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:14.391544 21237 raft_consensus.cc:603] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:13:14.392386 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:14.392979 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:14.393527 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:14.394362 21235 raft_consensus.cc:2379] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 3: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:14.395236 21150 raft_consensus.cc:1261] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:13:14.376389 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:14.398937 21320 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:13.956562 21151 raft_consensus.cc:1226] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 4. Ops: []
W20260812 19:13:14.373054 22008 leader_election.cc:334] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.775s (SENT)
W20260812 19:13:14.021512 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 301062us
W20260812 19:13:14.282692 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 316793us
I20260812 19:13:14.403704 21204 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:14.405139 21112 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:14.374054 21158 raft_consensus.cc:2379] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 3: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:14.379979 21328 raft_consensus.cc:2454] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:14.380990 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:14.380820 21327 raft_consensus.cc:2454] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:14.377386 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 229) took 1093 ms. Trace:
I20260812 19:13:13.963626 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:14.407959 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 218) took 1507 ms. Trace:
I20260812 19:13:14.408504 21156 raft_consensus.cc:2379] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 4: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:14.393642 21237 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 1.
I20260812 19:13:14.395562 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:14.400689 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:14.397433 21239 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:13:14.410408 21323 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:14.085034 22048 raft_consensus.cc:2379] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 4: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:14.411785 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:14.038729 23402 log.cc:924] Time spent T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.314s user 0.000s sys 0.001s
I20260812 19:13:14.408460 21327 rpcz_store.cc:276] 0812 19:13:12.900261 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:13.664518 (+764257us) service_pool.cc:227] Handling call
0812 19:13:14.407875 (+743357us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:14.141167 23404 log.cc:924] Time spent T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.531s user 0.000s sys 0.001s
I20260812 19:13:13.936997 22046 raft_consensus.cc:1226] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
W20260812 19:13:14.203866 23409 log.cc:924] Time spent T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.092s user 0.000s sys 0.001s
I20260812 19:13:13.945089 23213 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:14.413118 21203 leader_election.cc:398] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4.
W20260812 19:13:14.184386 23398 log.cc:924] Time spent T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.578s user 0.001s sys 0.001s
W20260812 19:13:14.129794 23418 log.cc:924] Time spent T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.174s user 0.000s sys 0.000s
W20260812 19:13:14.305073 23372 consensus_meta.cc:287] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.459s user 0.014s sys 0.001s
I20260812 19:13:14.114307 23422 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:14.293110 23370 consensus_meta.cc:287] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.928s user 0.011s sys 0.000s
I20260812 19:13:13.945303 23288 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:14.141713 23303 consensus_meta.cc:287] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.751s user 0.009s sys 0.004s
W20260812 19:13:14.293761 23209 consensus_meta.cc:287] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.451s user 0.012s sys 0.002s
I20260812 19:13:14.419037 21323 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
W20260812 19:13:14.111156 23395 log.cc:924] Time spent T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.456s user 0.001s sys 0.001s
W20260812 19:13:14.035621 23407 log.cc:924] Time spent T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
W20260812 19:13:14.294418 23176 consensus_meta.cc:287] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.371s user 0.013s sys 0.002s
W20260812 19:13:14.025310 23283 consensus_meta.cc:287] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.177s user 0.010s sys 0.006s
W20260812 19:13:14.033281 23417 log.cc:924] Time spent T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
W20260812 19:13:14.197539 23419 log.cc:924] Time spent T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20260812 19:13:14.416354 21114 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:14.037050 23406 log.cc:924] Time spent T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
W20260812 19:13:14.301079 23375 consensus_meta.cc:287] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.467s user 0.010s sys 0.004s
W20260812 19:13:14.039644 23411 log.cc:924] Time spent T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
I20260812 19:13:14.093271 23173 raft_consensus.cc:3032] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Stepping down as leader of term 3
W20260812 19:13:14.374267 23287 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.365s user 0.019s sys 0.020s
I20260812 19:13:14.423113 21203 leader_election.cc:401] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20260812 19:13:14.425421 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 52453us
I20260812 19:13:14.422436 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:14.039923 23412 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:14.181631 23335 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:14.277938 23269 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:14.146552 23423 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:14.301754 23313 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:14.330655 23369 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:14.507432 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 132717us
I20260812 19:13:14.415800 21329 rpcz_store.cc:276] 0812 19:13:13.283574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:13.889814 (+606240us) service_pool.cc:227] Handling call
0812 19:13:14.377294 (+487480us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:14.142267 23385 consensus_queue.cc:237] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:14.292096 23382 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:14.426672 21235 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:13:14.513713 21235 raft_consensus.cc:595] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Received request to transfer leadership to TS f741102bfe1441d1b453e5a838649059
I20260812 19:13:14.514140 21235 raft_consensus.cc:603] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:13:14.512842 21203 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:14.095762 23351 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:13.961387 23390 raft_consensus.cc:686] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:14.403172 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:14.127764 23364 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:14.412881 23376 consensus_meta.cc:287] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.048s user 0.009s sys 0.004s
I20260812 19:13:14.441854 21204 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:14.451856 21238 raft_consensus.cc:1261] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:13:14.447692 23372 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:14.462225 22011 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.030s sys 0.042s
I20260812 19:13:14.465771 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:13.941697 23199 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:14.442979 23274 raft_consensus.cc:2741] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260812 19:13:14.001844 23304 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:14.459563 21154 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
W20260812 19:13:14.453104 22047 consensus_meta.cc:287] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.096s user 0.014s sys 0.000s
I20260812 19:13:14.330325 23347 consensus_queue.cc:1046] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.129s
I20260812 19:13:14.497730 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1652) took 1832 ms. Trace:
W20260812 19:13:14.498926 22008 leader_election.cc:334] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.889s (SENT)
W20260812 19:13:14.493064 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 77435.7us
I20260812 19:13:14.451825 23422 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:14.445429 23208 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260812 19:13:14.001638 23392 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:14.431818 21239 raft_consensus.cc:1226] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:14.422379 23428 consensus_queue.cc:1035] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.082s
I20260812 19:13:14.174361 23220 consensus_queue.cc:237] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:14.097532 23160 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:14.516381 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 93054.6us
I20260812 19:13:14.516433 23196 consensus_queue.cc:1035] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:14.492992 23348 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:14.491027 23173 raft_consensus.cc:729] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:14.425295 23213 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:14.499570 23176 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:14.510933 23400 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.779s user 0.032s sys 0.037s
W20260812 19:13:14.538012 22011 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.686s user 0.030s sys 0.042s
I20260812 19:13:14.538125 21323 raft_consensus.cc:3037] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:14.645344 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 986182us
I20260812 19:13:14.454092 23370 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:14.463299 23303 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:14.540712 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:14.658697 21327 raft_consensus.cc:2379] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 4: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:14.537782 23351 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:14.463001 23209 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:14.463763 23288 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:14.549199 21203 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:14.511389 23269 raft_consensus.cc:2726] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4.
I20260812 19:13:14.552702 21236 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1403) took 1715 ms. Trace:
I20260812 19:13:14.510047 23424 raft_consensus.cc:2726] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:14.513553 23313 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:14.556818 22051 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40614 (request call id 383) took 1739 ms. Trace:
I20260812 19:13:14.497346 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1650) took 1833 ms. Trace:
I20260812 19:13:14.559379 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1647) took 1909 ms. Trace:
I20260812 19:13:14.475680 23283 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:14.556156 21324 rpcz_store.cc:276] 0812 19:13:12.664932 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.778915 (+113983us) service_pool.cc:227] Handling call
0812 19:13:12.779215 (+ 300us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:12.793137 (+ 13922us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:12.793176 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:12.793753 (+ 577us) log.cc:842] Serialized 26 byte log entry
0812 19:13:13.056081 (+262328us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:13.056118 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:13.056194 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:14.497500 (+1441306us) raft_consensus.cc:1671] finished
0812 19:13:14.497518 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:14.497662 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20611,"raft.queue_time_us":13639,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":17827,"thread_start_us":157585,"threads_started":1,"wal-append.queue_time_us":718970}
W20260812 19:13:14.563704 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 70901us
I20260812 19:13:14.508327 23412 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:14.491595 23375 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:14.594467 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 189441us
I20260812 19:13:14.606004 21200 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:14.555683 22047 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:14.610847 21238 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:13:14.611224 22053 raft_consensus.cc:2379] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 3: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
W20260812 19:13:14.611748 21149 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1659) took 2413 ms (client timeout 1531 ms). Trace:
I20260812 19:13:14.539310 23382 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:14.612263 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1406) took 1785 ms. Trace:
W20260812 19:13:14.564170 23366 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.300s user 0.013s sys 0.022s
I20260812 19:13:14.681273 22051 rpcz_store.cc:276] 0812 19:13:12.817457 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.817640 (+ 183us) service_pool.cc:227] Handling call
0812 19:13:12.835191 (+ 17551us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:12.893255 (+ 58064us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:12.893290 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:12.912903 (+ 19613us) log.cc:842] Serialized 26 byte log entry
0812 19:13:13.440097 (+527194us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:13.440154 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:13.440272 (+ 118us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:14.556583 (+1116311us) raft_consensus.cc:1671] finished
0812 19:13:14.556602 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:14.556756 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11226,"raft.queue_time_us":33598,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":17282,"thread_start_us":419013,"threads_started":1,"wal-append.queue_time_us":772627}
I20260812 19:13:14.616302 23429 raft_consensus.cc:2726] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:13:14.616912 23389 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:14.620939 23372 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:14.623118 23426 consensus_queue.cc:1046] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20260812 19:13:14.624140 23430 consensus_queue.cc:1035] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20260812 19:13:14.759706 23389 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:14.762741 22047 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40624 (request call id 326) took 1934 ms (client timeout 1505 ms). Trace:
I20260812 19:13:14.762962 23426 raft_consensus.cc:3032] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:14.747929 21243 rpcz_store.cc:276] 0812 19:13:12.826426 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.826530 (+ 104us) service_pool.cc:227] Handling call
0812 19:13:12.839595 (+ 13065us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:12.923545 (+ 83950us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:12.923605 (+ 60us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:13.021824 (+ 98219us) log.cc:842] Serialized 26 byte log entry
0812 19:13:13.568533 (+546709us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:13.568594 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:13.568712 (+ 118us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:14.612011 (+1043299us) raft_consensus.cc:1671] finished
0812 19:13:14.612033 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:14.612170 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":85673,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":82637,"thread_start_us":419516,"threads_started":2,"wal-append.queue_time_us":596981}
I20260812 19:13:14.763659 23426 raft_consensus.cc:729] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:13:14.763305 22047 rpcz_store.cc:269] 0812 19:13:12.819878 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.827614 (+ 7736us) service_pool.cc:227] Handling call
0812 19:13:14.754325 (+1926711us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:14.765148 23426 consensus_queue.cc:260] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:14.625051 23433 raft_consensus.cc:2741] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260812 19:13:14.514720 23369 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:14.652410 22056 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Advancing to term 3
W20260812 19:13:14.652935 23380 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.008s sys 0.010s
I20260812 19:13:14.542464 23393 consensus_queue.cc:1046] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.129s
W20260812 19:13:14.770953 23380 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s user 0.008s sys 0.011s
I20260812 19:13:14.771700 23393 raft_consensus.cc:3032] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:14.772408 23393 raft_consensus.cc:729] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:14.773327 23393 consensus_queue.cc:260] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:14.603832 23347 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:14.774914 23347 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:14.511300 23423 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:14.784705 23423 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:14.664654 21236 rpcz_store.cc:276] 0812 19:13:12.825514 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.825665 (+ 151us) service_pool.cc:227] Handling call
0812 19:13:12.825991 (+ 326us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:13.031194 (+205203us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:13.031243 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:13.070140 (+ 38897us) log.cc:842] Serialized 26 byte log entry
0812 19:13:13.612791 (+542651us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:13.612844 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:13.612974 (+ 130us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:14.540434 (+927460us) raft_consensus.cc:1671] finished
0812 19:13:14.540458 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:14.540613 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":18103,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"thread_start_us":514187,"threads_started":2,"wal-append.queue_time_us":733469}
I20260812 19:13:14.678115 23445 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:14.553905 23364 consensus_queue.cc:1035] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.094s
I20260812 19:13:14.646374 23334 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:14.539682 23376 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:14.573643 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 131916us
I20260812 19:13:14.683182 23400 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260812 19:13:14.745504 22008 leader_election.cc:334] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.920s (SENT)
I20260812 19:13:14.813375 22008 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:14.648255 23348 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:14.805759 23334 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:14.717420 21327 raft_consensus.cc:1261] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:13:14.748190 21238 raft_consensus.cc:595] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Received request to transfer leadership to TS f741102bfe1441d1b453e5a838649059
W20260812 19:13:14.751430 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 145563us
W20260812 19:13:14.748447 21149 rpcz_store.cc:269] 0812 19:13:12.198555 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.203455 (+ 4900us) service_pool.cc:227] Handling call
0812 19:13:14.611447 (+2407992us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x7b5400324954. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:14.611639 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1145435520}
I20260812 19:13:14.751945 21323 raft_consensus.cc:2454] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:14.747611 21112 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:14.759886 23428 raft_consensus.cc:2781] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:14.752489 21322 rpcz_store.cc:276] 0812 19:13:12.650368 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.651019 (+ 651us) service_pool.cc:227] Handling call
0812 19:13:12.651470 (+ 451us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:12.732579 (+ 81109us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:12.732626 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:12.757573 (+ 24947us) log.cc:842] Serialized 26 byte log entry
0812 19:13:13.265286 (+507713us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:13.265356 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:13.265456 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:14.559024 (+1293568us) raft_consensus.cc:1671] finished
0812 19:13:14.559060 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:14.559272 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":24469,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":26965,"thread_start_us":367268,"threads_started":2,"wal-append.queue_time_us":587844}
W20260812 19:13:14.780844 23427 log.cc:924] Time spent T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.646s user 0.002s sys 0.006s
I20260812 19:13:14.780857 23283 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:14.781360 22056 raft_consensus.cc:1261] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEARNER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:13:14.781545 23347 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:14.666702 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 117586us
I20260812 19:13:14.863286 21323 raft_consensus.cc:1261] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:13:14.865106 21323 raft_consensus.cc:1261] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:13:14.770285 23369 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:14.791998 23199 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:14.740038 23439 log.cc:924] Time spent T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
W20260812 19:13:14.746956 23366 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.493s user 0.018s sys 0.023s
I20260812 19:13:14.803246 23393 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:14.808362 21204 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:14.615660 23320 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:14.725494 23432 raft_consensus.cc:3032] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:14.800101 23382 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:14.723066 23435 raft_consensus.cc:2781] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:14.690466 21153 raft_consensus.cc:1226] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:14.697824 21241 raft_consensus.cc:2379] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 4: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:14.696789 23381 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:14.778313 23173 consensus_queue.cc:260] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:14.681433 21318 rpcz_store.cc:276] 0812 19:13:12.664209 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.712853 (+ 48644us) service_pool.cc:227] Handling call
0812 19:13:12.732057 (+ 19204us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:12.745637 (+ 13580us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:12.745670 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:12.757094 (+ 11424us) log.cc:842] Serialized 26 byte log entry
0812 19:13:13.341828 (+584734us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:13.341869 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:13.341940 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:14.497102 (+1155162us) raft_consensus.cc:1671] finished
0812 19:13:14.497125 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:14.497273 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13185,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":8315,"thread_start_us":569776,"threads_started":1,"wal-append.queue_time_us":807951}
I20260812 19:13:14.858671 21238 raft_consensus.cc:603] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:13:14.860428 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1382) took 2031 ms. Trace:
I20260812 19:13:14.923449 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:14.924394 21328 raft_consensus.cc:2379] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:14.923772 21327 rpcz_store.cc:276] 0812 19:13:12.828996 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:14.666991 (+1837995us) service_pool.cc:227] Handling call
0812 19:13:14.716878 (+ 49887us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:14.859143 (+142265us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:14.860334 (+ 1191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:14.658514 23176 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:14.697602 23313 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:14.871254 21323 raft_consensus.cc:1226] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260812 19:13:14.820948 23400 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:14.876307 21321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 214) took 2005 ms (client timeout 2017 ms). Trace:
I20260812 19:13:14.815088 23376 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:14.815150 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 69728.9us
I20260812 19:13:14.860119 23389 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:14.904971 21322 raft_consensus.cc:1261] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260812 19:13:14.861093 23428 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:14.861071 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 113531us
I20260812 19:13:14.870759 23436 raft_consensus.cc:2781] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:14.778028 23346 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:14.872368 23426 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:14.933537 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1659) took 1003 ms. Trace:
W20260812 19:13:14.976572 22008 leader_election.cc:334] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.970s (SENT)
W20260812 19:13:14.939738 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 131483us
I20260812 19:13:14.977156 21323 rpcz_store.cc:276] 0812 19:13:13.929917 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:14.867737 (+937820us) service_pool.cc:227] Handling call
0812 19:13:14.869560 (+ 1823us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:14.933135 (+ 63575us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:14.933458 (+ 323us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:14.977632 21204 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412
W20260812 19:13:14.931581 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 326450us
I20260812 19:13:14.821964 23445 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:14.978947 21112 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:14.811339 23391 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:14.920428 23435 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:14.876014 23088 consensus_queue.cc:1046] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.099s
I20260812 19:13:14.899744 23432 raft_consensus.cc:729] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:14.898248 23304 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:14.821883 23372 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 19:13:14.938822 21321 rpcz_store.cc:269] 0812 19:13:12.857842 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.875884 (+ 18042us) service_pool.cc:227] Handling call
0812 19:13:14.862475 (+1986591us) spinlock_profiling.cc:243] Waited 826 ms on lock 0x7b540004f1d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:14.862654 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1899944704}
I20260812 19:13:14.976420 23436 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:14.821700 23364 consensus_queue.cc:1035] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.156s
I20260812 19:13:14.980037 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1388) took 1078 ms. Trace:
I20260812 19:13:14.695385 23390 consensus_queue.cc:237] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:14.980341 21320 consensus_meta.cc:287] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.567s user 0.003s sys 0.008s
W20260812 19:13:14.936488 21287 connection.cc:232] RPC call timeout handler was delayed by -3.0695940069854259s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15100000000000002s for a response
I20260812 19:13:14.954252 23209 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:14.981161 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1392) took 1042 ms. Trace:
I20260812 19:13:14.982460 21112 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:14.991504 22008 leader_election.cc:334] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.017s (SENT)
I20260812 19:13:14.991652 21238 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:13:14.995829 21322 rpcz_store.cc:276] 0812 19:13:13.901240 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:14.862796 (+961556us) service_pool.cc:227] Handling call
0812 19:13:14.904432 (+ 41636us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:14.979679 (+ 75247us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:14.979958 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:14.998004 21241 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1404) took 2172 ms (client timeout 2137 ms). Trace:
I20260812 19:13:15.000442 21153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1663) took 2779 ms. Trace:
I20260812 19:13:15.014516 21244 raft_consensus.cc:1226] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 3. Ops: []
W20260812 19:13:15.014287 21287 connection.cc:232] RPC call timeout handler was delayed by -2.9939940070034936s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15940000000000001s for a response
I20260812 19:13:14.994800 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:14.987850 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1661) took 1057 ms. Trace:
I20260812 19:13:15.014240 21320 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:15.007069 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1662) took 1076 ms. Trace:
I20260812 19:13:14.815346 23208 consensus_queue.cc:1035] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.164s
I20260812 19:13:14.932735 23334 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:15.024652 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1405) took 2198 ms. Trace:
W20260812 19:13:15.151795 21115 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.027s sys 0.061s
I20260812 19:13:15.103328 21155 raft_consensus.cc:1226] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
W20260812 19:13:15.140933 21114 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.011s sys 0.039s
W20260812 19:13:15.235970 21287 connection.cc:232] RPC call timeout handler was delayed by -2.9894940070807934s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15990000000000001s for a response
W20260812 19:13:15.079427 21203 connection.cc:232] RPC call timeout handler was delayed by -1.473802226013504s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1531s for a response
W20260812 19:13:15.235965 21115 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.669s user 0.029s sys 0.070s
I20260812 19:13:14.944994 23400 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:15.244236 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1658) took 1314 ms. Trace:
I20260812 19:13:15.244834 21320 rpcz_store.cc:276] 0812 19:13:13.929591 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:14.375990 (+446399us) service_pool.cc:227] Handling call
0812 19:13:15.244147 (+868157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:15.246287 21153 rpcz_store.cc:276] 0812 19:13:12.220854 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.225750 (+ 4896us) service_pool.cc:227] Handling call
0812 19:13:12.282826 (+ 57076us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:14.920605 (+2637779us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:15.000065 (+ 79460us) spinlock_profiling.cc:243] Waited 2.41 s on lock 0x7b5400215554. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:15.000298 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5536421632}
I20260812 19:13:15.101260 21157 raft_consensus.cc:1226] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
W20260812 19:13:15.240298 21241 rpcz_store.cc:269] 0812 19:13:12.825832 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.826368 (+ 536us) service_pool.cc:227] Handling call
0812 19:13:14.997627 (+2171259us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b54002567d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:14.997841 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4154280832}
W20260812 19:13:15.250186 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 267791us
I20260812 19:13:15.239457 21321 raft_consensus.cc:3037] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 4
W20260812 19:13:15.113888 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 127919us
W20260812 19:13:15.239274 21287 connection.cc:232] RPC call timeout handler was delayed by -2.9228940070606768s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1673s for a response
W20260812 19:13:15.239097 21114 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.669s user 0.014s sys 0.047s
W20260812 19:13:15.080431 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 136637us
W20260812 19:13:15.110427 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 132856us
I20260812 19:13:15.236555 21327 rpcz_store.cc:276] 0812 19:13:13.930795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:14.935920 (+1005125us) service_pool.cc:227] Handling call
0812 19:13:15.006304 (+ 70384us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:15.006585 (+ 281us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:15.006646 (+ 61us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:15.006702 (+ 56us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:15.006753 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:15.006824 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:15.006983 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:15.239638 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 681021us
I20260812 19:13:15.241101 21323 rpcz_store.cc:276] 0812 19:13:13.938702 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:14.978030 (+1039328us) service_pool.cc:227] Handling call
0812 19:13:14.978356 (+ 326us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:14.978548 (+ 192us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:14.978571 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:14.980765 (+ 2194us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:14.980846 (+ 81us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:14.980936 (+ 90us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:14.981083 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:15.242040 21318 rpcz_store.cc:276] 0812 19:13:13.930450 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:14.932444 (+1001994us) service_pool.cc:227] Handling call
0812 19:13:14.987295 (+ 54851us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:14.987497 (+ 202us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:14.987522 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:14.987563 (+ 41us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:14.987601 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:14.987668 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:14.987783 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:15.272979 21204 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:15.243161 21329 raft_consensus.cc:1226] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260812 19:13:14.949352 23351 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:15.235750 21240 rpcz_store.cc:276] 0812 19:13:12.826126 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.826452 (+ 326us) service_pool.cc:227] Handling call
0812 19:13:12.839051 (+ 12599us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:12.874508 (+ 35457us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:12.874539 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:12.875036 (+ 497us) log.cc:842] Serialized 26 byte log entry
0812 19:13:13.319376 (+444340us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:13.319422 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:13.319529 (+ 107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:15.012260 (+1692731us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b240011d938. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d69e2cf4 00007f52d69e0dfb 00007f52d6a6f559 00007f52d6a6f1e5 00007f52d6a6e740 00007f52d9755578 00007f52d9755cd8 00007f52d972d6e8 00007f52d9728aea 00007f52d9727f36
0812 19:13:15.013812 (+ 1552us) raft_consensus.cc:1671] finished
0812 19:13:15.013837 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:15.024542 (+ 10705us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48492800,"thread_start_us":406666,"threads_started":1,"wal-append.queue_time_us":911838}
W20260812 19:13:15.274925 21287 connection.cc:232] RPC call timeout handler was delayed by -2.8733940070960671s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1728s for a response
I20260812 19:13:14.975698 23380 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:15.278728 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:15.280227 21287 connection.cc:232] RPC call timeout handler was delayed by -2.865294007002376s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17370000000000002s for a response
W20260812 19:13:15.285744 21287 connection.cc:232] RPC call timeout handler was delayed by -2.864394007017836s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1738s for a response
W20260812 19:13:15.286118 21287 connection.cc:232] RPC call timeout handler was delayed by -2.6249940070556477s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20040000000000002s for a response
I20260812 19:13:15.001442 23257 raft_consensus.cc:3032] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Stepping down as leader of term 4
W20260812 19:13:15.009605 23420 log.cc:924] Time spent T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.875s user 0.000s sys 0.008s
I20260812 19:13:14.983174 23428 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:15.287655 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:14.994381 23436 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:15.011741 23412 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:15.294184 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 302693us
I20260812 19:13:15.306998 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:14.982738 23376 raft_consensus.cc:2781] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:15.002722 22050 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40614 (request call id 380) took 2489 ms. Trace:
I20260812 19:13:15.295104 21320 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:14.986325 23435 raft_consensus.cc:3037] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:15.100986 23320 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:15.291749 22011 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:15.295186 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 44068.7us
I20260812 19:13:15.082695 23370 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:15.296411 23455 consensus_queue.cc:1035] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260812 19:13:15.301240 21327 raft_consensus.cc:1261] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:13:15.301702 23443 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:15.303119 23351 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:15.113777 23269 raft_consensus.cc:2781] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:15.303722 23366 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:15.300282 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:15.364982 21239 raft_consensus.cc:2379] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 4: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:15.304548 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:15.305625 23348 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:15.121306 23088 consensus_queue.cc:1225] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer f741102bfe1441d1b453e5a838649059 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260812 19:13:14.988298 23303 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:15.254899 23375 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:14.992082 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:14.989876 23288 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:15.315052 21200 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:15.379846 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 64901.3us
I20260812 19:13:15.277727 23391 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:15.385782 21239 raft_consensus.cc:1261] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260812 19:13:15.385593 23391 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:15.110597 23392 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:15.386936 22045 raft_consensus.cc:2354] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 2. Current term is 3.
W20260812 19:13:15.387504 22045 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40624 (request call id 327) took 1748 ms (client timeout 1599 ms). Trace:
W20260812 19:13:15.388085 22045 rpcz_store.cc:269] 0812 19:13:13.639238 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:13.651294 (+ 12056us) service_pool.cc:227] Handling call
0812 19:13:14.991887 (+1340593us) spinlock_profiling.cc:243] Waited 9 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:13:15.387414 (+395527us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20701568}
W20260812 19:13:15.297739 21287 connection.cc:232] RPC call timeout handler was delayed by -2.6123940070392564s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20179999999999998s for a response
I20260812 19:13:15.319751 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:15.390571 21287 connection.cc:232] RPC call timeout handler was delayed by -2.5934940070146695s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20390000000000003s for a response
W20260812 19:13:15.391129 21287 connection.cc:232] RPC call timeout handler was delayed by -2.577294007060118s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2057s for a response
I20260812 19:13:15.391412 21154 raft_consensus.cc:2396] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260812 19:13:14.980664 23449 raft_consensus.cc:2726] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:15.312749 23389 consensus_queue.cc:1035] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260812 19:13:15.322077 21235 raft_consensus.cc:1226] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:15.322697 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:15.322748 21115 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:15.323545 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:15.324048 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:15.408066 22047 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:15.324702 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:15.326184 21236 raft_consensus.cc:1226] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260812 19:13:15.282406 23385 raft_consensus.cc:2781] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:15.319041 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:15.410919 21319 raft_consensus.cc:2379] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 4: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
W20260812 19:13:15.411509 21319 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1663) took 1480 ms (client timeout 1641 ms). Trace:
W20260812 19:13:15.412014 21319 rpcz_store.cc:269] 0812 19:13:13.931159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.294858 (+1363699us) service_pool.cc:227] Handling call
0812 19:13:15.411427 (+116569us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:15.329080 21320 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:15.289143 23430 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:15.339673 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:15.339781 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:15.334942 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:15.415232 21151 raft_consensus.cc:2379] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:15.292733 23257 raft_consensus.cc:729] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:14.992741 23432 consensus_queue.cc:260] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:15.349131 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 57460.3us
I20260812 19:13:15.350039 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:15.452102 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1385) took 2622 ms. Trace:
I20260812 19:13:15.355895 21233 raft_consensus.cc:1261] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:13:15.357460 21149 raft_consensus.cc:1261] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260812 19:13:14.993981 23213 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:15.309124 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:15.312511 23347 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:15.318979 23209 raft_consensus.cc:2781] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:13:15.315038 23376 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:15.366252 21322 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:15.364077 23269 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:15.294654 23380 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:15.321614 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:15.361205 23445 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:15.391503 21287 connection.cc:232] RPC call timeout handler was delayed by -2.5394940070109442s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20990000000000003s for a response
I20260812 19:13:15.407088 23442 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:15.408543 23370 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:15.376575 23351 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:15.427290 21321 raft_consensus.cc:2454] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:15.318819 22050 rpcz_store.cc:276] 0812 19:13:12.513368 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.570707 (+ 57339us) service_pool.cc:227] Handling call
0812 19:13:12.594341 (+ 23634us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:14.399956 (+1805615us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:15.002275 (+602319us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b540064afd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:15.002568 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3024571648}
I20260812 19:13:15.397070 23313 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:15.351406 23412 consensus_queue.cc:1035] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260812 19:13:15.448727 23422 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:15.393920 23348 consensus_queue.cc:1035] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20260812 19:13:15.367487 23382 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:15.413869 23430 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:15.346030 23443 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:15.705139 23443 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:15.368047 23349 consensus_queue.cc:1035] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.083s
I20260812 19:13:15.711186 23430 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:15.365868 23372 raft_consensus.cc:2781] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:15.379016 23366 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:15.452759 21324 rpcz_store.cc:276] 0812 19:13:12.829501 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:14.744144 (+1914643us) service_pool.cc:227] Handling call
0812 19:13:14.744570 (+ 426us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:14.747114 (+ 2544us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:14.747144 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:14.747595 (+ 451us) log.cc:842] Serialized 26 byte log entry
0812 19:13:15.276849 (+529254us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:15.276886 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:15.276955 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:15.451798 (+174843us) raft_consensus.cc:1671] finished
0812 19:13:15.451827 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:15.452024 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":187623,"raft.run_cpu_time_us":3439,"raft.run_wall_time_us":38891,"thread_start_us":460031,"threads_started":1,"wal-append.queue_time_us":603546}
I20260812 19:13:15.689299 23376 raft_consensus.cc:3037] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:15.570101 22053 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40614 (request call id 381) took 2927 ms (client timeout 1526 ms). Trace:
I20260812 19:13:15.505805 23385 raft_consensus.cc:686] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:15.690768 23451 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:15.753611 23451 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:15.690475 23370 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:15.691660 21287 connection.cc:232] RPC call timeout handler was delayed by -2.4179940070025623s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22340000000000002s for a response
I20260812 19:13:15.526281 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:15.505506 23459 raft_consensus.cc:2726] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:15.692840 23213 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:15.692866 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:15.766283 23334 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:15.767167 22045 raft_consensus.cc:2379] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 3: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:15.767463 23451 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:15.693723 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:15.768492 21319 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:15.769184 23391 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:15.769062 23385 consensus_queue.cc:237] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:15.632189 21244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37894 (request call id 1460) took 2037 ms. Trace:
I20260812 19:13:15.383337 23176 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:15.694446 23442 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:15.633841 21155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 202) took 2341 ms. Trace:
I20260812 19:13:15.776281 21155 rpcz_store.cc:276] 0812 19:13:13.292879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:13.678917 (+386038us) service_pool.cc:227] Handling call
0812 19:13:13.778788 (+ 99871us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:15.276470 (+1497682us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:15.633453 (+356983us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400215a54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:15.633748 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3040160000}
I20260812 19:13:15.690141 21152 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:15.454350 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 131755us
I20260812 19:13:15.675807 23173 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:15.701273 23209 raft_consensus.cc:686] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:15.343149 23320 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:15.788929 23459 consensus_queue.cc:1035] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:13:15.790904 23206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31c6a2accba4d29ba096500b9d1a911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 19:13:15.734369 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:15.737879 22055 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 336) took 1084 ms. Trace:
I20260812 19:13:15.739765 21157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 197) took 2467 ms. Trace:
I20260812 19:13:15.746645 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1381) took 2917 ms. Trace:
I20260812 19:13:15.794819 21329 rpcz_store.cc:276] 0812 19:13:12.828783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:14.515303 (+1686520us) service_pool.cc:227] Handling call
0812 19:13:15.242922 (+727619us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:15.280171 (+ 37249us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:15.746335 (+466164us) spinlock_profiling.cc:243] Waited 699 ms on lock 0x7b540016e4d0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:15.746536 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1608299776}
I20260812 19:13:15.791842 23213 consensus_queue.cc:1046] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260812 19:13:15.804220 23459 raft_consensus.cc:3032] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:15.804692 23459 raft_consensus.cc:729] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:15.805956 23459 consensus_queue.cc:260] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:15.792163 21322 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:15.811184 21115 leader_election.cc:398] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Vote denied by peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579) with higher term. Message: Invalid argument: T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 2. Current term is 3.
I20260812 19:13:15.504516 23389 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:15.758361 21233 raft_consensus.cc:1261] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260812 19:13:15.765132 21287 connection.cc:232] RPC call timeout handler was delayed by -2.4062940070871264s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2247s for a response
W20260812 19:13:15.825368 21287 connection.cc:232] RPC call timeout handler was delayed by -1.4783940070774406s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32780000000000004s for a response
I20260812 19:13:15.767058 22011 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:15.392822 23088 raft_consensus.cc:2781] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:15.770527 21244 rpcz_store.cc:276] 0812 19:13:13.594745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:13.594935 (+ 190us) service_pool.cc:227] Handling call
0812 19:13:15.014327 (+1419392us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:15.240869 (+226542us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:15.631854 (+390985us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54002562d0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:15.632082 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3263200000}
I20260812 19:13:15.637641 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:15.558035 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:15.751473 23347 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:15.782953 21151 raft_consensus.cc:1226] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260812 19:13:15.784155 23442 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:15.828646 21149 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:15.813192 23209 consensus_queue.cc:237] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:15.815050 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:15.829865 21155 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:15.815783 21240 raft_consensus.cc:1261] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260812 19:13:15.794535 21157 rpcz_store.cc:276] 0812 19:13:13.272365 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:13.272602 (+ 237us) service_pool.cc:227] Handling call
0812 19:13:13.273064 (+ 462us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:15.248441 (+1975377us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:15.739403 (+490962us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b5400194bd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:15.739683 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4152098688}
I20260812 19:13:15.759130 23372 raft_consensus.cc:686] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:15.792958 22055 rpcz_store.cc:276] 0812 19:13:14.653297 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:14.653455 (+ 158us) service_pool.cc:227] Handling call
0812 19:13:14.653829 (+ 374us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:14.774999 (+121170us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:14.775046 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:14.775091 (+ 45us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:14.775131 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:15.737510 (+962379us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b5400452254. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:15.737572 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:15.737770 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":24838,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":6789,"spinlock_wait_cycles":277781248}
W20260812 19:13:15.750087 22053 rpcz_store.cc:269] 0812 19:13:12.642707 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:12.642893 (+ 186us) service_pool.cc:227] Handling call
0812 19:13:15.511810 (+2868917us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b540061fb54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:15.569971 (+ 58161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4490738048}
I20260812 19:13:15.824519 21115 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:15.827905 21154 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:15.856683 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:15.857697 21244 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:15.768507 23451 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:15.854445 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:15.776937 23269 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:15.864737 22055 raft_consensus.cc:2362] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3. Re-sending same reply.
I20260812 19:13:15.873786 21329 raft_consensus.cc:1226] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260812 19:13:15.878123 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 121677us
I20260812 19:13:15.880054 22011 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:15.859985 21287 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:15.845772 23459 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:15.885218 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:15.887640 21150 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:15.889994 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:15.908872 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:15.909796 22050 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:15.863708 23254 raft_consensus.cc:2781] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260812 19:13:15.918367 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 51795.5us
I20260812 19:13:15.922194 21115 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:15.858582 23428 consensus_meta.cc:287] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.559s user 0.008s sys 0.005s
I20260812 19:13:15.923254 22053 raft_consensus.cc:2454] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:15.741884 23257 consensus_queue.cc:260] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:15.929111 21115 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:15.917542 22011 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:15.908102 21240 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:15.873618 23313 raft_consensus.cc:2781] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 19:13:15.940467 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:15.903041 23435 consensus_meta.cc:287] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.555s user 0.006s sys 0.006s
W20260812 19:13:15.969250 21328 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s user 0.017s sys 0.034s
W20260812 19:13:15.969566 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 109714us
I20260812 19:13:15.970711 21240 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
W20260812 19:13:15.974170 21328 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s user 0.017s sys 0.034s
I20260812 19:13:15.889171 23088 raft_consensus.cc:686] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:15.977639 23088 consensus_queue.cc:237] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:15.907260 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:15.905530 23422 raft_consensus.cc:2726] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:13:15.918944 23366 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:15.919201 23455 raft_consensus.cc:2741] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260812 19:13:15.863550 23389 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:15.938076 23255 raft_consensus.cc:2781] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
W20260812 19:13:16.012905 22008 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.022s sys 0.025s
I20260812 19:13:15.938099 23443 raft_consensus.cc:2781] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:15.881507 23451 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:15.982024 23429 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.738s user 0.018s sys 0.034s
I20260812 19:13:15.997624 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:16.049830 21240 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
W20260812 19:13:16.033910 22008 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.726s user 0.022s sys 0.025s
W20260812 19:13:16.051127 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 743698us
I20260812 19:13:16.052462 21244 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:15.968289 23320 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:16.058177 23390 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.756s user 0.035s sys 0.016s
I20260812 19:13:16.060788 22053 raft_consensus.cc:3032] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:16.061694 21150 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:15.989099 23257 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:15.920543 23014 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.937s user 0.026s sys 0.035s
I20260812 19:13:16.071832 21287 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:15.938150 23428 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:16.169448 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:16.151481 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:16.203738 22053 raft_consensus.cc:729] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
W20260812 19:13:16.044602 23381 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.055s user 0.025s sys 0.051s
I20260812 19:13:16.031589 23225 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:16.043187 23443 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:16.028375 23350 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.708s user 0.024s sys 0.036s
I20260812 19:13:16.188136 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:16.122722 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 45343.7us
W20260812 19:13:16.262499 21112 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.929s user 0.008s sys 0.032s
W20260812 19:13:16.263685 21112 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.930s user 0.008s sys 0.032s
W20260812 19:13:16.263990 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 930602us
W20260812 19:13:16.062592 23429 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.827s user 0.018s sys 0.035s
I20260812 19:13:16.204133 21240 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:16.233923 21323 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.883s user 0.013s sys 0.019s
I20260812 19:13:15.985278 23435 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:16.269999 21323 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.919s user 0.013s sys 0.019s
W20260812 19:13:16.051913 23421 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.868s user 0.023s sys 0.090s
I20260812 19:13:16.260617 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:16.278509 21235 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:16.062331 23370 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:16.267843 21236 raft_consensus.cc:1226] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20260812 19:13:16.123660 23436 consensus_meta.cc:287] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.798s user 0.008s sys 0.005s
W20260812 19:13:16.203163 23390 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.901s user 0.038s sys 0.024s
I20260812 19:13:16.207145 21152 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:16.209985 22053 consensus_queue.cc:260] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:16.295526 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:16.261575 23350 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.948s user 0.032s sys 0.036s
W20260812 19:13:16.278119 23421 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.096s user 0.024s sys 0.097s
I20260812 19:13:16.143360 21149 raft_consensus.cc:2454] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
W20260812 19:13:16.282189 23393 consensus_meta.cc:287] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.396s user 0.013s sys 0.001s
I20260812 19:13:16.313123 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:16.283126 23254 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:16.282840 23349 consensus_queue.cc:1035] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260812 19:13:16.257790 23381 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.260s user 0.027s sys 0.062s
I20260812 19:13:16.073235 23380 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:16.062356 23255 raft_consensus.cc:686] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:16.187423 23320 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:16.251034 23014 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 1.267s user 0.027s sys 0.035s
W20260812 19:13:16.260488 23258 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.994s user 0.025s sys 0.025s
I20260812 19:13:16.309743 21238 raft_consensus.cc:595] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:13:16.310267 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:16.263669 23422 raft_consensus.cc:2726] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:16.270865 23443 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:16.323158 23254 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:16.314540 21329 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
W20260812 19:13:16.314435 21327 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.969s user 0.013s sys 0.036s
I20260812 19:13:16.283643 23428 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:16.294737 22053 raft_consensus.cc:3037] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:16.285396 23351 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:16.325826 21233 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20260812 19:13:16.353906 21233 pending_rounds.cc:76] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Aborting all ops after (but not including) 3
I20260812 19:13:16.354259 21233 pending_rounds.cc:97] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20260812 19:13:16.354564 21233 raft_consensus.cc:2866] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:16.331467 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:16.269181 23225 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:16.311568 23370 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:16.337733 21238 raft_consensus.cc:603] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:13:16.167785 23455 raft_consensus.cc:2781] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:16.379604 23455 raft_consensus.cc:686] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:16.380987 23455 consensus_queue.cc:237] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:16.338124 21237 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:16.388604 23389 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:16.061470 23372 consensus_queue.cc:237] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:16.313122 23425 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:16.349654 21327 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.004s user 0.013s sys 0.036s
I20260812 19:13:16.323689 23371 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:16.337855 23347 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:16.308777 23436 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:16.341797 23258 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.094s user 0.025s sys 0.026s
I20260812 19:13:16.359002 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:16.416952 21324 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.662s user 0.005s sys 0.014s
W20260812 19:13:16.417510 21324 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.663s user 0.005s sys 0.014s
I20260812 19:13:16.418205 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:16.365280 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 332019us
I20260812 19:13:16.412935 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:16.330094 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:16.367425 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:16.431149 21235 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:16.377236 23225 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:16.426357 21150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1685) took 1117 ms. Trace:
W20260812 19:13:16.427706 21322 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.009s sys 0.012s
I20260812 19:13:16.428066 21200 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:16.437327 21150 rpcz_store.cc:276] 0812 19:13:15.308912 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.309037 (+ 125us) service_pool.cc:227] Handling call
0812 19:13:16.426096 (+1117059us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b54003cfd54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:16.426266 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1088424704}
W20260812 19:13:16.333029 23438 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.019s sys 0.026s
I20260812 19:13:16.438480 21200 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:16.439700 21200 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:16.440277 21287 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
W20260812 19:13:16.440829 21322 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.618s user 0.009s sys 0.012s
I20260812 19:13:16.340497 23320 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:16.445415 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:16.445472 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:16.449223 21200 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:16.449625 22056 raft_consensus.cc:2454] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:16.450654 21156 raft_consensus.cc:1226] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: []
W20260812 19:13:16.451083 21319 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.681s user 0.000s sys 0.045s
W20260812 19:13:16.451440 21319 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.681s user 0.000s sys 0.045s
W20260812 19:13:16.441365 23173 consensus_meta.cc:287] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.633s user 0.010s sys 0.002s
I20260812 19:13:16.466491 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:16.471290 21235 raft_consensus.cc:2379] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:16.458061 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:16.622179 23432 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:16.441844 23438 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.746s user 0.019s sys 0.026s
I20260812 19:13:16.622903 23347 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:16.628499 21237 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:16.447247 23176 raft_consensus.cc:2781] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:16.632098 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:16.622555 23436 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:16.629714 22053 raft_consensus.cc:1261] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:16.447641 23364 raft_consensus.cc:2781] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:16.692447 21329 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:16.692593 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:16.651752 22008 leader_election.cc:334] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.603s (SENT)
I20260812 19:13:16.409883 23371 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:16.409659 23425 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:16.395468 23370 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:16.444355 23255 consensus_queue.cc:237] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:16.463029 23428 raft_consensus.cc:2781] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:16.632897 23208 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:17.322999 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:16.912274 22051 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.530s user 0.000s sys 0.023s
W20260812 19:13:17.355327 22051 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.973s user 0.000s sys 0.026s
W20260812 19:13:16.840598 21241 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.235s user 0.016s sys 0.006s
W20260812 19:13:17.015588 21320 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.278s user 0.011s sys 0.042s
W20260812 19:13:17.356554 21320 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.619s user 0.013s sys 0.043s
W20260812 19:13:17.100061 21240 consensus_meta.cc:287] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.830s user 0.011s sys 0.000s
W20260812 19:13:16.765851 22055 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.800s user 0.008s sys 0.011s
W20260812 19:13:17.357899 22055 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.392s user 0.009s sys 0.016s
I20260812 19:13:16.462661 23393 raft_consensus.cc:2781] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
W20260812 19:13:17.015960 21318 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.621s user 0.019s sys 0.004s
W20260812 19:13:17.323040 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 671287us
W20260812 19:13:17.315447 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 599065us
W20260812 19:13:16.923295 21155 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.055s user 0.013s sys 0.031s
W20260812 19:13:17.367951 22008 leader_election.cc:334] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.710s (SENT)
W20260812 19:13:17.368269 21155 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.500s user 0.020s sys 0.032s
W20260812 19:13:16.631392 23472 log.cc:924] Time spent T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.231s user 0.000s sys 0.000s
W20260812 19:13:16.913533 21203 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.645s user 0.001s sys 0.025s
W20260812 19:13:17.301352 23014 log.cc:924] Time spent T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.197s user 0.001s sys 0.000s
W20260812 19:13:16.765636 22011 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.799s user 0.008s sys 0.011s
I20260812 19:13:17.359225 21240 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
W20260812 19:13:17.376927 22011 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.411s user 0.010s sys 0.020s
I20260812 19:13:17.377154 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1421) took 1226 ms. Trace:
W20260812 19:13:17.377271 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 1.4597e+06us
I20260812 19:13:17.377540 21240 rpcz_store.cc:276] 0812 19:13:16.150484 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:16.151019 (+ 535us) service_pool.cc:227] Handling call
0812 19:13:17.377083 (+1226064us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:17.359735 21241 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.754s user 0.018s sys 0.009s
I20260812 19:13:17.378243 22011 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:17.378412 22008 leader_election.cc:334] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.637s (SENT)
I20260812 19:13:16.697396 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:17.269666 23376 consensus_meta.cc:287] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.502s user 0.010s sys 0.004s
W20260812 19:13:17.371317 21203 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 2.103s user 0.001s sys 0.033s
W20260812 19:13:17.108649 23467 log.cc:924] Time spent T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.656s user 0.001s sys 0.000s
W20260812 19:13:17.380802 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 2.11259e+06us
W20260812 19:13:17.314237 23269 consensus_meta.cc:287] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.424s user 0.008s sys 0.003s
I20260812 19:13:17.388386 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:17.372601 21237 raft_consensus.cc:2379] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 6: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
W20260812 19:13:16.763725 23468 log.cc:924] Time spent T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
I20260812 19:13:16.623900 23451 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:16.454205 23426 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.732s user 0.006s sys 0.014s
I20260812 19:13:16.658315 23176 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:17.145727 23476 log.cc:924] Time spent T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.183s user 0.000s sys 0.000s
W20260812 19:13:17.225351 23400 consensus_meta.cc:287] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.935s user 0.011s sys 0.005s
W20260812 19:13:16.871826 23313 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.841s user 0.013s sys 0.105s
W20260812 19:13:17.225188 23209 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.367s user 0.013s sys 0.004s
W20260812 19:13:17.413775 23385 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.033s user 0.012s sys 0.041s
W20260812 19:13:17.368813 21318 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.974s user 0.022s sys 0.004s
I20260812 19:13:17.379043 22008 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:17.390733 21151 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.022s user 0.009s sys 0.008s
W20260812 19:13:17.434731 21151 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.066s user 0.009s sys 0.009s
I20260812 19:13:16.835912 23364 raft_consensus.cc:686] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:17.145534 23017 tablet_copy_client.cc:531] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260812 19:13:17.024160 23429 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.011s sys 0.028s
I20260812 19:13:16.834965 23435 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:17.409561 21112 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.091s user 0.009s sys 0.011s
I20260812 19:13:17.345273 23428 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:17.444842 23385 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.064s user 0.013s sys 0.042s
I20260812 19:13:17.347719 23422 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:17.368798 23380 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:17.393324 23392 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.033s user 0.023s sys 0.042s
I20260812 19:13:17.448364 23473 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:17.394856 23376 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:17.394973 23269 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:17.376037 21204 connection.cc:232] RPC call timeout handler was delayed by -3.157646361971274s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.207s for a response
W20260812 19:13:17.391050 21321 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.465s user 0.007s sys 0.013s
I20260812 19:13:16.698640 23347 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:17.386284 23320 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:16.913055 23254 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:17.417295 23426 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.687s user 0.012s sys 0.020s
I20260812 19:13:17.367268 23436 consensus_queue.cc:1035] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20260812 19:13:17.453182 21329 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:17.454396 21156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1682) took 2146 ms. Trace:
W20260812 19:13:17.456997 21112 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.138s user 0.010s sys 0.011s
W20260812 19:13:17.438936 23313 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.408s user 0.013s sys 0.115s
I20260812 19:13:17.372045 23258 consensus_queue.cc:1046] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
I20260812 19:13:17.411634 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:17.458835 21288 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.029s user 0.035s sys 0.040s
I20260812 19:13:17.448927 23400 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:17.417814 23451 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:17.373535 23208 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:17.425943 23176 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:17.453853 23429 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.142s user 0.011s sys 0.033s
W20260812 19:13:17.466872 23088 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.207s user 0.015s sys 0.011s
W20260812 19:13:17.461896 22045 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.083s user 0.000s sys 0.020s
W20260812 19:13:17.467150 21200 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.016s user 0.008s sys 0.010s
I20260812 19:13:17.419607 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:17.432770 23209 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.575s user 0.014s sys 0.014s
W20260812 19:13:17.419898 23381 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.067s user 0.007s sys 0.010s
W20260812 19:13:17.461376 21244 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.211s user 0.004s sys 0.020s
W20260812 19:13:17.480425 21321 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.555s user 0.008s sys 0.014s
W20260812 19:13:17.919725 23459 consensus_meta.cc:287] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 2.013s user 0.018s sys 0.000s
W20260812 19:13:17.906872 23504 log.cc:924] Time spent T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.273s user 0.000s sys 0.001s
W20260812 19:13:17.920496 23501 log.cc:924] Time spent T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.307s user 0.000s sys 0.001s
I20260812 19:13:17.466080 23428 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:17.894640 22011 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.000s sys 0.009s
W20260812 19:13:17.954185 22011 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.575s user 0.001s sys 0.009s
I20260812 19:13:17.714046 21155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1692) took 2392 ms. Trace:
W20260812 19:13:17.708037 21319 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.256s user 0.007s sys 0.015s
W20260812 19:13:17.635888 21202 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.604s user 0.003s sys 0.019s
W20260812 19:13:17.479331 21324 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.037s user 0.020s sys 0.038s
I20260812 19:13:17.486641 23435 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
W20260812 19:13:17.948688 21288 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.519s user 0.038s sys 0.047s
W20260812 19:13:17.519819 21322 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.076s user 0.020s sys 0.040s
W20260812 19:13:17.715135 23482 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.014s sys 0.050s
W20260812 19:13:17.894783 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 516384us
W20260812 19:13:17.970041 22055 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.006s sys 0.019s
W20260812 19:13:17.975741 22055 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s user 0.006s sys 0.019s
W20260812 19:13:17.975831 22008 leader_election.cc:334] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.642s (SENT)
W20260812 19:13:17.971417 21322 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.527s user 0.020s sys 0.044s
W20260812 19:13:17.976673 22008 leader_election.cc:334] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.677s (SENT)
I20260812 19:13:17.977156 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1677) took 2575 ms. Trace:
I20260812 19:13:17.545073 21158 raft_consensus.cc:2379] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 4: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:13:17.947602 21200 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.496s user 0.013s sys 0.015s
W20260812 19:13:17.474864 23392 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.121s user 0.024s sys 0.047s
W20260812 19:13:17.978305 21114 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.006s sys 0.019s
W20260812 19:13:17.978781 21114 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.568s user 0.006s sys 0.019s
W20260812 19:13:17.978600 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 1.52734e+06us
W20260812 19:13:17.979487 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 569027us
W20260812 19:13:17.948300 21329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1410) took 1627 ms (client timeout 1724 ms). Trace:
I20260812 19:13:17.980304 21200 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:17.481091 21204 connection.cc:232] RPC call timeout handler was delayed by -2.8843576218932867s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2001s for a response
W20260812 19:13:17.950076 21320 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.001s sys 0.019s
I20260812 19:13:17.977556 21322 rpcz_store.cc:276] 0812 19:13:15.401905 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.821060 (+419155us) service_pool.cc:227] Handling call
0812 19:13:15.821603 (+ 543us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:15.821883 (+ 280us) raft_consensus.cc:1520] Early marking committed up to index 6
0812 19:13:16.444044 (+622161us) log.cc:842] Serialized 14 byte log entry
0812 19:13:17.976795 (+1532751us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:17.976846 (+ 51us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:17.976893 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:17.976956 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:17.977093 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15981,"prepare.queue_time_us":1509392,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"thread_start_us":2154067,"threads_started":2}
W20260812 19:13:17.982210 21320 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.625s user 0.001s sys 0.019s
I20260812 19:13:17.983393 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1398) took 3557 ms. Trace:
I20260812 19:13:17.983906 21320 rpcz_store.cc:276] 0812 19:13:14.426213 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.735932 (+1309719us) service_pool.cc:227] Handling call
0812 19:13:15.736333 (+ 401us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:15.736631 (+ 298us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:17.357083 (+1620452us) log.cc:842] Serialized 14 byte log entry
0812 19:13:17.982858 (+625775us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:17.982944 (+ 86us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:17.983018 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:17.983122 (+ 104us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:17.983309 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":715330,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"thread_start_us":2245184,"threads_started":2}
I20260812 19:13:17.986064 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:17.948789 21233 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
W20260812 19:13:17.987314 21204 connection.cc:232] RPC call timeout handler was delayed by -2.4133032680256292s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20350000000000001s for a response
I20260812 19:13:17.987866 21233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1438) took 1234 ms. Trace:
I20260812 19:13:17.988430 21233 rpcz_store.cc:276] 0812 19:13:16.753617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:17.370413 (+616796us) service_pool.cc:227] Handling call
0812 19:13:17.987764 (+617351us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:17.486702 21287 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.039s user 0.000s sys 0.019s
W20260812 19:13:17.962705 21319 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.511s user 0.011s sys 0.022s
W20260812 19:13:17.962844 21202 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.933s user 0.011s sys 0.025s
W20260812 19:13:17.950048 21244 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.700s user 0.004s sys 0.026s
I20260812 19:13:17.950320 21156 rpcz_store.cc:276] 0812 19:13:15.307896 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.308056 (+ 160us) service_pool.cc:227] Handling call
0812 19:13:15.347166 (+ 39110us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:16.451139 (+1103973us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:17.454087 (+1002948us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400214654. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:17.454321 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2533309184}
W20260812 19:13:17.962939 21324 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.521s user 0.028s sys 0.056s
I20260812 19:13:17.948596 21240 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
W20260812 19:13:17.990011 21287 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.542s user 0.002s sys 0.030s
W20260812 19:13:17.675889 23487 log.cc:924] Time spent T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
W20260812 19:13:17.958542 21203 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.009s sys 0.014s
W20260812 19:13:18.000372 21203 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.616s user 0.009s sys 0.014s
W20260812 19:13:17.971236 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 1.54152e+06us
W20260812 19:13:17.963024 23393 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.011s sys 0.011s
W20260812 19:13:17.980361 21329 rpcz_store.cc:269] 0812 19:13:16.321138 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:16.631812 (+310674us) service_pool.cc:227] Handling call
0812 19:13:17.948223 (+1316411us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:17.519388 21115 consensus_peers.cc:487] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260812 19:13:17.963344 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 585114us
I20260812 19:13:17.990537 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1439) took 1040 ms. Trace:
I20260812 19:13:17.493722 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1666) took 3066 ms. Trace:
I20260812 19:13:17.722452 23494 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:17.888764 23480 log.cc:924] Time spent T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.164s user 0.000s sys 0.000s
I20260812 19:13:18.006691 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:17.669447 23490 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election won for term 3
W20260812 19:13:18.025408 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 578381us
W20260812 19:13:17.993391 21237 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.008s sys 0.030s
W20260812 19:13:17.996680 21320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1414) took 1638 ms (client timeout 1803 ms). Trace:
I20260812 19:13:18.024000 21323 rpcz_store.cc:276] 0812 19:13:14.413680 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.285353 (+871673us) service_pool.cc:227] Handling call
0812 19:13:15.285817 (+ 464us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:15.286282 (+ 465us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:15.286321 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:15.286925 (+ 604us) log.cc:842] Serialized 26 byte log entry
0812 19:13:16.270520 (+983595us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:16.270572 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:16.270694 (+ 122us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:17.479476 (+1208782us) raft_consensus.cc:1671] finished
0812 19:13:17.479496 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:17.479674 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":920,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":29784,"thread_start_us":919289,"threads_started":1,"wal-append.queue_time_us":1388346}
W20260812 19:13:18.027366 21320 rpcz_store.cc:269] 0812 19:13:16.358852 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:17.985667 (+1626815us) service_pool.cc:227] Handling call
0812 19:13:17.996597 (+ 10930us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:17.963184 23429 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:17.723476 23491 raft_consensus.cc:2726] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:13:17.997153 21243 raft_consensus.cc:2379] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 3: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:17.652351 23515 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:17.962777 21155 rpcz_store.cc:276] 0812 19:13:15.321685 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.858129 (+536444us) service_pool.cc:227] Handling call
0812 19:13:15.867452 (+ 9323us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:15.867735 (+ 283us) raft_consensus.cc:1520] Early marking committed up to index 2
0812 19:13:17.368791 (+1501056us) log.cc:842] Serialized 14 byte log entry
0812 19:13:17.713502 (+344711us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:17.713580 (+ 78us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:17.713650 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:17.713760 (+ 110us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:17.713955 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1798359,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":221212,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":7013,"thread_start_us":1844776,"threads_started":2}
W20260812 19:13:17.950783 23257 consensus_meta.cc:287] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.742s user 0.010s sys 0.006s
W20260812 19:13:17.956158 23455 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.021s user 0.004s sys 0.021s
W20260812 19:13:17.949651 23088 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.690s user 0.024s sys 0.024s
W20260812 19:13:17.953495 23381 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.601s user 0.007s sys 0.028s
I20260812 19:13:17.972173 21112 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:17.613083 23371 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:17.578953 23364 consensus_queue.cc:237] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:18.068996 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1667) took 3649 ms. Trace:
I20260812 19:13:17.655359 23425 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:18.052903 21150 raft_consensus.cc:1226] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260812 19:13:17.710923 23376 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:17.945431 23370 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:18.004499 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 1.56418e+06us
W20260812 19:13:17.949797 22045 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.571s user 0.001s sys 0.029s
I20260812 19:13:18.023473 21329 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:18.082823 23529 raft_consensus.cc:2781] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:18.083307 23529 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:17.953284 23313 raft_consensus.cc:686] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
W20260812 19:13:18.005228 21204 leader_election.cc:334] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Timed out: RequestConsensusVote RPC to 127.19.70.132:39579 timed out after 2.070s (SENT)
W20260812 19:13:17.977859 22008 leader_election.cc:334] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.720s (SENT)
W20260812 19:13:18.093070 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 115233us
W20260812 19:13:18.094129 22008 leader_election.cc:334] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.857s (SENT)
I20260812 19:13:18.023994 21240 rpcz_store.cc:276] 0812 19:13:16.963605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:17.411215 (+447610us) service_pool.cc:227] Handling call
0812 19:13:17.990461 (+579246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:18.028277 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:18.094551 22008 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:18.076399 23500 log.cc:924] Time spent T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.372s user 0.000s sys 0.001s
I20260812 19:13:17.876979 23347 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:18.101441 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:18.071624 23269 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:18.030375 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:18.031296 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:18.042316 23490 raft_consensus.cc:686] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:17.967023 23477 raft_consensus.cc:2726] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:17.481815 23380 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:18.022037 23393 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.612s user 0.013s sys 0.011s
W20260812 19:13:17.975157 23482 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.841s user 0.018s sys 0.055s
I20260812 19:13:18.036798 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:18.037914 21233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1436) took 1588 ms. Trace:
I20260812 19:13:17.474850 23422 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:18.039585 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:18.022394 23372 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.016s sys 0.036s
W20260812 19:13:18.178737 23372 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.850s user 0.016s sys 0.037s
I20260812 19:13:18.180100 23372 raft_consensus.cc:2726] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:17.998200 23209 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:18.180922 23209 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:18.181341 23209 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:18.181900 23524 log.cc:924] Time spent T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.097s user 0.001s sys 0.003s
I20260812 19:13:18.069703 23491 consensus_queue.cc:1046] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20260812 19:13:18.183179 23491 consensus_queue.cc:1225] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer f741102bfe1441d1b453e5a838649059 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260812 19:13:18.116812 23313 consensus_queue.cc:237] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:18.049799 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:18.003335 23502 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:18.052767 23429 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:17.962759 23451 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:17.492549 23473 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:18.032823 23014 log.cc:924] Time spent T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
W20260812 19:13:18.032766 23506 log.cc:924] Time spent T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.368s user 0.002s sys 0.000s
I20260812 19:13:17.991770 23435 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:18.051930 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:18.075177 21318 rpcz_store.cc:276] 0812 19:13:14.419793 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.285175 (+865382us) service_pool.cc:227] Handling call
0812 19:13:15.313383 (+ 28208us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:15.347688 (+ 34305us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:15.347733 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:15.348289 (+ 556us) log.cc:842] Serialized 26 byte log entry
0812 19:13:17.433480 (+2085191us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:17.433538 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:17.433643 (+ 105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:18.068750 (+635107us) raft_consensus.cc:1671] finished
0812 19:13:18.068773 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:18.068923 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9310,"raft.queue_time_us":20278,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":21478,"thread_start_us":2038268,"threads_started":1,"wal-append.queue_time_us":2301714}
I20260812 19:13:18.080843 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:18.076996 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 104935us
I20260812 19:13:18.266146 21155 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:18.079252 21241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1412) took 2664 ms. Trace:
I20260812 19:13:18.084779 21287 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:18.083389 22046 raft_consensus.cc:3032] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:18.095170 21320 raft_consensus.cc:2454] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
W20260812 19:13:18.001873 23478 log.cc:924] Time spent T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.324s user 0.000s sys 0.001s
W20260812 19:13:18.071197 23443 consensus_meta.cc:287] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.710s user 0.016s sys 0.000s
W20260812 19:13:18.081573 23497 log.cc:924] Time spent T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.366s user 0.000s sys 0.001s
I20260812 19:13:18.083737 23529 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:18.097563 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:18.050107 23533 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:18.080127 23455 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.149s user 0.009s sys 0.026s
W20260812 19:13:18.092389 23508 log.cc:924] Time spent T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.403s user 0.001s sys 0.001s
I20260812 19:13:18.067301 23254 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:18.030310 21237 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.586s user 0.009s sys 0.030s
I20260812 19:13:18.103575 21322 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:18.105063 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:18.107671 21236 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1413) took 2501 ms. Trace:
I20260812 19:13:18.103581 21323 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:18.081967 23371 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:18.092869 23530 consensus_queue.cc:1035] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
W20260812 19:13:18.112588 23507 log.cc:924] Time spent T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.408s user 0.000s sys 0.001s
I20260812 19:13:18.028971 23494 raft_consensus.cc:504] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:18.111819 23393 raft_consensus.cc:686] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:18.112694 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1670) took 3133 ms. Trace:
I20260812 19:13:18.092309 23370 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:18.138016 22053 raft_consensus.cc:2396] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260812 19:13:18.114495 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 109662us
I20260812 19:13:18.137786 21233 rpcz_store.cc:276] 0812 19:13:16.449493 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:17.997252 (+1547759us) service_pool.cc:227] Handling call
0812 19:13:18.028460 (+ 31208us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:18.033579 (+ 5119us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:18.033644 (+ 65us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:18.037536 (+ 3892us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:18.037610 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:18.037691 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:18.037851 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4725,"raft.queue_time_us":73986,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":113}
I20260812 19:13:18.174593 23400 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:18.177511 23176 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:18.187460 21329 raft_consensus.cc:2396] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260812 19:13:18.187556 21152 raft_consensus.cc:2379] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:18.216462 23436 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:18.116353 23498 log.cc:924] Time spent T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.414s user 0.000s sys 0.002s
I20260812 19:13:18.052525 23459 consensus_queue.cc:1035] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.094s
W20260812 19:13:18.174593 23522 log.cc:924] Time spent T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
W20260812 19:13:18.251658 23432 consensus_meta.cc:287] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.597s user 0.013s sys 0.004s
I20260812 19:13:18.052315 23515 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:18.326191 21112 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:18.326582 21322 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1407) took 2031 ms (client timeout 1768 ms). Trace:
I20260812 19:13:18.326681 21153 raft_consensus.cc:2396] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260812 19:13:18.303622 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34764 (request call id 22) took 3312 ms. Trace:
I20260812 19:13:18.327581 21238 rpcz_store.cc:276] 0812 19:13:14.991288 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:14.991456 (+ 168us) service_pool.cc:227] Handling call
0812 19:13:18.303280 (+3311824us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540018fe54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa0bc3 00007f52de9ea81a 00007f52d80e9d04 00007f52d80e9c92 00007f52d80e9bc1 00007f52d80e9b3c 00007f52d80e8df2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:18.303478 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2458524928}
I20260812 19:13:18.327244 21321 rpcz_store.cc:276] 0812 19:13:14.979053 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.778922 (+799869us) service_pool.cc:227] Handling call
0812 19:13:15.823912 (+ 44990us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:15.859285 (+ 35373us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:15.859329 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:15.883211 (+ 23882us) log.cc:842] Serialized 26 byte log entry
0812 19:13:17.946430 (+2063219us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:17.946687 (+ 257us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:17.946797 (+ 110us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:18.112440 (+165643us) raft_consensus.cc:1671] finished
0812 19:13:18.112462 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:18.112631 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":35067,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":23769,"thread_start_us":2020489,"threads_started":1,"wal-append.queue_time_us":1757524}
I20260812 19:13:18.307479 21151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1691) took 2989 ms. Trace:
I20260812 19:13:18.329314 21154 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
W20260812 19:13:18.324743 21323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1405) took 2038 ms (client timeout 2101 ms). Trace:
I20260812 19:13:18.336454 21237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1431) took 1975 ms. Trace:
W20260812 19:13:18.336680 21329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1406) took 2047 ms (client timeout 1927 ms). Trace:
W20260812 19:13:18.337324 21329 rpcz_store.cc:269] 0812 19:13:16.289634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.006356 (+1716722us) service_pool.cc:227] Handling call
0812 19:13:18.336552 (+330196us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:18.340997 21322 rpcz_store.cc:269] 0812 19:13:16.295256 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:17.982820 (+1687564us) service_pool.cc:227] Handling call
0812 19:13:18.326492 (+343672us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:18.325327 21236 rpcz_store.cc:276] 0812 19:13:15.606631 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.624716 (+ 18085us) service_pool.cc:227] Handling call
0812 19:13:15.625252 (+ 536us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:16.288042 (+662790us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:18.103119 (+1815077us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x7b54003bea54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:18.107589 (+ 4470us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1476909824}
I20260812 19:13:18.343628 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:18.334317 21323 rpcz_store.cc:269] 0812 19:13:16.286797 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.027829 (+1741032us) service_pool.cc:227] Handling call
0812 19:13:18.324645 (+296816us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:18.348388 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:18.349359 21323 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:13:18.352891 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:18.353286 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:18.354190 21318 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:18.249648 23372 raft_consensus.cc:3032] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:18.355522 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:18.356330 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 271629us
I20260812 19:13:18.329154 21151 rpcz_store.cc:276] 0812 19:13:15.318050 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.858199 (+540149us) service_pool.cc:227] Handling call
0812 19:13:15.858589 (+ 390us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:15.867859 (+ 9270us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:15.867888 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:15.868447 (+ 559us) log.cc:842] Serialized 26 byte log entry
0812 19:13:16.368515 (+500068us) spinlock_profiling.cc:243] Waited 1.95 ms on lock 0x7b1800289a20. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b6c345 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b89e22 00007f52d97275bb 00007f52d972d063 00007f52d972d411 00007f52d9a7a5cf
0812 19:13:17.435597 (+1067082us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:17.435646 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:17.435744 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:18.307084 (+871340us) raft_consensus.cc:1671] finished
0812 19:13:18.307106 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:18.307397 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3266,"raft.queue_time_us":420,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":141,"spinlock_wait_cycles":4478848,"thread_start_us":1066789,"threads_started":1,"wal-append.queue_time_us":1700032}
I20260812 19:13:18.337150 21237 rpcz_store.cc:276] 0812 19:13:16.365566 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:17.410315 (+1044749us) service_pool.cc:227] Handling call
0812 19:13:17.444090 (+ 33775us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:18.335900 (+891810us) raft_consensus.cc:1520] Early marking committed up to index 1
0812 19:13:18.335933 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:18.335992 (+ 59us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:18.336047 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:18.336159 (+ 112us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:18.336352 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":502844,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":9818,"thread_start_us":886310,"threads_started":1}
I20260812 19:13:18.267138 21241 rpcz_store.cc:276] 0812 19:13:15.415342 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.416133 (+ 791us) service_pool.cc:227] Handling call
0812 19:13:15.447301 (+ 31168us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:15.447862 (+ 561us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:15.447902 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:15.448446 (+ 544us) log.cc:842] Serialized 26 byte log entry
0812 19:13:17.378689 (+1930243us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:17.378729 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:17.378802 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:18.078999 (+700197us) raft_consensus.cc:1671] finished
0812 19:13:18.079021 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:18.079186 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":45008,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":99300,"thread_start_us":1772796,"threads_started":1,"wal-append.queue_time_us":2078621}
W20260812 19:13:18.289367 23535 log.cc:924] Time spent T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.217s user 0.003s sys 0.000s
I20260812 19:13:18.361092 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1400) took 3614 ms. Trace:
I20260812 19:13:18.295522 23451 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:18.292866 23429 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:18.375563 23494 leader_election.cc:288] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:18.377629 23176 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:18.378630 21158 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1693) took 3056 ms (client timeout 1547 ms). Trace:
I20260812 19:13:18.377689 23473 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:18.379120 23515 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:18.357451 23533 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:18.356007 23334 raft_consensus.cc:2726] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:18.381762 23533 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:18.357313 23370 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:18.351142 23243 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:18.360026 23385 raft_consensus.cc:2781] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:18.383893 23385 raft_consensus.cc:686] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:18.383775 23243 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:18.384827 23370 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:18.363248 21287 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:18.385025 23385 consensus_queue.cc:237] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:18.385681 23206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:40624 (request call id 289) took 12758 ms. Trace:
I20260812 19:13:18.386499 21287 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:18.387815 21287 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:18.388561 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260812 19:13:18.389654 21287 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:18.351398 23400 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:18.363538 23393 consensus_queue.cc:237] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:18.365064 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:18.390952 21203 connection.cc:232] RPC call timeout handler was delayed by -1.1508471891283989s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
I20260812 19:13:18.391191 21321 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:18.360108 22046 raft_consensus.cc:729] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:18.366974 21323 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 1.
I20260812 19:13:18.367877 23208 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:18.371028 21329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1415) took 2007 ms (client timeout 2293 ms). Trace:
I20260812 19:13:18.380946 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:18.382943 22055 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 345) took 2485 ms. Trace:
W20260812 19:13:18.391361 21203 connection.cc:232] RPC call timeout handler was delayed by -1.1238471891265363s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1547s for a response
W20260812 19:13:18.379130 21158 rpcz_store.cc:269] 0812 19:13:15.322029 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.323264 (+ 1235us) service_pool.cc:227] Handling call
0812 19:13:18.378376 (+3055112us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b5400214b54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:18.378513 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4924476160}
I20260812 19:13:18.394399 22051 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 337) took 3118 ms. Trace:
I20260812 19:13:18.394688 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260812 19:13:18.324769 21320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1421) took 1870 ms (client timeout 2021 ms). Trace:
W20260812 19:13:18.366117 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 257806us
I20260812 19:13:18.362165 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1399) took 3616 ms. Trace:
I20260812 19:13:18.400205 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:18.398947 23451 raft_consensus.cc:2726] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:18.479590 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:18.397981 23335 raft_consensus.cc:2726] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 19:13:18.485885 21203 connection.cc:232] RPC call timeout handler was delayed by -1.0959471890237182s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15780000000000002s for a response
W20260812 19:13:18.527551 21203 connection.cc:232] RPC call timeout handler was delayed by -1.0392471890663728s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16410000000000002s for a response
W20260812 19:13:18.481172 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 479850us
I20260812 19:13:18.529093 21288 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:13:18.396391 23334 raft_consensus.cc:2726] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:18.488579 21323 raft_consensus.cc:2379] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 4: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:18.396536 23490 consensus_queue.cc:237] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:18.486697 21329 rpcz_store.cc:269] 0812 19:13:16.363774 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.338589 (+1974815us) service_pool.cc:227] Handling call
0812 19:13:18.370949 (+ 32360us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:18.339465 23436 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:18.479854 23370 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:18.380970 23014 tablet_bootstrap.cc:492] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260812 19:13:18.491394 21320 rpcz_store.cc:269] 0812 19:13:16.454887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.028023 (+1573136us) service_pool.cc:227] Handling call
0812 19:13:18.324693 (+296670us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:18.478442 22051 rpcz_store.cc:276] 0812 19:13:15.276317 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.277289 (+ 972us) service_pool.cc:227] Handling call
0812 19:13:15.296068 (+ 18779us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:15.325598 (+ 29530us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:15.325629 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:15.326064 (+ 435us) log.cc:842] Serialized 26 byte log entry
0812 19:13:17.355883 (+2029819us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:17.355940 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:17.356040 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:18.394080 (+1038040us) raft_consensus.cc:1671] finished
0812 19:13:18.394109 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:18.394313 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":30213,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":27315,"thread_start_us":1973904,"threads_started":1,"wal-append.queue_time_us":2195918}
I20260812 19:13:18.562705 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:18.361584 23372 raft_consensus.cc:729] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:18.285465 23422 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:18.492094 21324 rpcz_store.cc:276] 0812 19:13:14.746183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.744020 (+997837us) service_pool.cc:227] Handling call
0812 19:13:15.753785 (+ 9765us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:15.754107 (+ 322us) raft_consensus.cc:1520] Early marking committed up to index 6
0812 19:13:16.441494 (+687387us) log.cc:842] Serialized 14 byte log entry
0812 19:13:18.361726 (+1920232us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:18.361793 (+ 67us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:18.361864 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:18.361951 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:18.362096 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14723,"prepare.queue_time_us":962167,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":62,"raft.queue_time_us":94038,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":11707,"thread_start_us":2582838,"threads_started":3,"wal-append.queue_time_us":1740979}
I20260812 19:13:18.401156 23176 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:18.508015 21318 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
W20260812 19:13:18.513549 21244 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.013s sys 0.033s
I20260812 19:13:18.390489 23528 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:18.519330 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 129731us
I20260812 19:13:18.372665 21319 rpcz_store.cc:276] 0812 19:13:14.747008 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.769229 (+1022221us) service_pool.cc:227] Handling call
0812 19:13:15.769646 (+ 417us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:15.769899 (+ 253us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:16.451866 (+681967us) log.cc:842] Serialized 14 byte log entry
0812 19:13:18.360676 (+1908810us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:18.360782 (+ 106us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:18.360848 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:18.360916 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:18.361032 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":251597,"prepare.queue_time_us":1191246,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":69,"raft.queue_time_us":90269,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":17875,"thread_start_us":2589817,"threads_started":3,"wal-append.queue_time_us":1644342}
I20260812 19:13:18.525476 21322 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
W20260812 19:13:18.570140 21322 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1417) took 2168 ms (client timeout 2588 ms). Trace:
W20260812 19:13:18.570654 21322 rpcz_store.cc:269] 0812 19:13:16.401440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.342052 (+1940612us) service_pool.cc:227] Handling call
0812 19:13:18.570066 (+228014us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:18.482365 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 449946us
I20260812 19:13:18.533754 21203 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:18.534423 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 175255us
I20260812 19:13:18.536680 21321 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:18.327735 23443 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:18.579558 23432 consensus_queue.cc:1035] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 19:13:18.580487 23443 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:18.254217 23502 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:18.542660 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:18.581874 21329 raft_consensus.cc:2454] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:18.541252 23477 raft_consensus.cc:2781] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:18.582962 23477 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:18.478053 22055 rpcz_store.cc:276] 0812 19:13:15.897413 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.914516 (+ 17103us) service_pool.cc:227] Handling call
0812 19:13:15.915055 (+ 539us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:17.358381 (+1443326us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:17.358425 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:17.389985 (+ 31560us) log.cc:842] Serialized 26 byte log entry
0812 19:13:17.976116 (+586131us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:17.976156 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:17.976251 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:18.288563 (+312312us) spinlock_profiling.cc:243] Waited 5.12 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:18.382568 (+ 94005us) raft_consensus.cc:1671] finished
0812 19:13:18.382598 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:18.382877 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":117002,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":23101,"spinlock_wait_cycles":11781888,"thread_start_us":1960082,"threads_started":2,"wal-append.queue_time_us":633061}
I20260812 19:13:18.538926 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1397) took 4116 ms. Trace:
I20260812 19:13:18.540392 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:18.351727 23371 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:18.494633 23257 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:18.567144 21244 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s user 0.013s sys 0.033s
W20260812 19:13:18.567853 21318 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1404) took 2281 ms (client timeout 2196 ms). Trace:
W20260812 19:13:18.607038 21318 rpcz_store.cc:269] 0812 19:13:16.286458 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.277856 (+1991398us) service_pool.cc:227] Handling call
0812 19:13:18.567691 (+289835us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:18.607481 21244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1420) took 2544 ms. Trace:
I20260812 19:13:18.608458 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:18.574728 23473 consensus_queue.cc:1035] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:13:18.575657 21114 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:18.583330 23477 raft_consensus.cc:3037] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:18.586300 22045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 347) took 2379 ms. Trace:
W20260812 19:13:18.587489 21243 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57682 (request call id 176) took 3288 ms (client timeout 1914 ms). Trace:
W20260812 19:13:18.632529 21243 rpcz_store.cc:269] 0812 19:13:15.299458 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.299567 (+ 109us) service_pool.cc:227] Handling call
0812 19:13:18.587209 (+3287642us) spinlock_profiling.cc:243] Waited 2.3 s on lock 0x7b5400041fd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:18.587393 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5283101056}
I20260812 19:13:18.586541 23372 consensus_queue.cc:260] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:18.538054 23542 raft_consensus.cc:2726] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:18.587846 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:18.641445 21324 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:18.589598 21112 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:18.583218 21329 raft_consensus.cc:1226] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
W20260812 19:13:18.643301 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 67728.7us
I20260812 19:13:18.643296 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:18.644101 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1403) took 2358 ms. Trace:
I20260812 19:13:18.644348 21322 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:18.644479 21329 rpcz_store.cc:276] 0812 19:13:16.286166 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.582678 (+2296512us) service_pool.cc:227] Handling call
0812 19:13:18.583029 (+ 351us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:18.643810 (+ 60781us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:18.644037 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:18.645746 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 56258.8us
I20260812 19:13:18.474913 23400 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:18.594625 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 60957.3us
I20260812 19:13:18.583137 23502 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:18.655539 22011 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.017s sys 0.017s
W20260812 19:13:18.656186 22011 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.614s user 0.017s sys 0.017s
I20260812 19:13:18.655292 21329 raft_consensus.cc:1261] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260812 19:13:18.656544 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:18.657841 22055 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:18.658094 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1413) took 2307 ms. Trace:
I20260812 19:13:18.595598 21323 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:18.549548 23243 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:18.665328 21318 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:13:18.595338 21327 rpcz_store.cc:276] 0812 19:13:14.422252 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.344591 (+922339us) service_pool.cc:227] Handling call
0812 19:13:15.345004 (+ 413us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:15.345234 (+ 230us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:18.382936 (+3037702us) spinlock_profiling.cc:243] Waited 2.74 ms on lock 0x7b1800248080. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d973582c 00007f52d6b6c9dc 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b87155 00007f52dd47b167 00007f52d9ab0807 00007f52d9abff0f
0812 19:13:18.383088 (+ 152us) log.cc:842] Serialized 14 byte log entry
0812 19:13:18.538184 (+155096us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:18.538294 (+ 110us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:18.538367 (+ 73us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:18.538523 (+ 156us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:18.538801 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6017,"prepare.queue_time_us":1055742,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":12765,"raft.queue_time_us":4865,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":10699,"spinlock_wait_cycles":6312704,"thread_start_us":1210589,"threads_started":2}
I20260812 19:13:18.599689 23429 raft_consensus.cc:2726] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:18.605870 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:18.607992 21244 rpcz_store.cc:276] 0812 19:13:16.063238 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:16.065182 (+ 1944us) service_pool.cc:227] Handling call
0812 19:13:16.065616 (+ 434us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:16.065902 (+ 286us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:17.993849 (+1927947us) log.cc:842] Serialized 14 byte log entry
0812 19:13:18.607040 (+613191us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:18.607103 (+ 63us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:18.607180 (+ 77us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:18.607265 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:18.607412 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":687853,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":19277,"thread_start_us":2356314,"threads_started":2}
I20260812 19:13:18.542984 23515 consensus_queue.cc:1046] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260812 19:13:18.590410 21287 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:18.658337 23385 raft_consensus.cc:2781] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:18.646672 21114 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:18.575730 23519 log.cc:924] Time spent T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.193s user 0.001s sys 0.000s
W20260812 19:13:18.656674 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 617460us
I20260812 19:13:18.641795 23257 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:18.660516 21329 rpcz_store.cc:276] 0812 19:13:16.350988 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.645268 (+2294280us) service_pool.cc:227] Handling call
0812 19:13:18.645596 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:18.657324 (+ 11728us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:18.658022 (+ 698us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:18.705072 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:18.705984 22051 raft_consensus.cc:2379] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:18.706920 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:18.707688 22056 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:18.708631 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:18.710172 22056 raft_consensus.cc:1226] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260812 19:13:18.711283 23546 log.cc:924] Time spent T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.141s user 0.000s sys 0.000s
W20260812 19:13:18.714068 23514 log.cc:924] Time spent T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.612s user 0.004s sys 0.009s
I20260812 19:13:18.706933 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:18.602972 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:18.715859 21320 raft_consensus.cc:2379] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:18.715826 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:18.666321 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1411) took 2339 ms. Trace:
I20260812 19:13:18.716763 22052 raft_consensus.cc:2379] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:18.716848 21318 rpcz_store.cc:276] 0812 19:13:16.326562 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.616981 (+2290419us) service_pool.cc:227] Handling call
0812 19:13:18.664813 (+ 47832us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:18.665943 (+ 1130us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:18.666240 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:18.671159 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:18.614575 23017 tablet_bootstrap.cc:492] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:13:18.386119 23206 rpcz_store.cc:276] 0812 19:13:05.627040 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:05.628708 (+ 1668us) service_pool.cc:227] Handling call
0812 19:13:06.481020 (+852312us) ts_tablet_manager.cc:928] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551)
0812 19:13:15.773126 (+9292106us) tablet_copy_client.cc:361] Tablet copy session begun
0812 19:13:17.880142 (+2107016us) tablet_metadata.cc:641] Metadata flushed
0812 19:13:17.918299 (+ 38157us) tablet_copy_client.cc:490] Wrote new tablet metadata
0812 19:13:18.384709 (+466410us) tablet_copy_client.cc:411] Wrote new consensus metadata
0812 19:13:18.385049 (+ 340us) tablet_replica.cc:205] Creating consensus instance
0812 19:13:18.385418 (+ 369us) ts_tablet_manager.cc:983] Replica registered.
0812 19:13:18.385535 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":30768,"tablet-copy.queue_time_us":358149,"thread_start_us":1574536,"threads_started":2}
I20260812 19:13:18.565622 23370 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:18.646572 21112 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:18.700750 22008 leader_election.cc:334] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.032s (SENT)
I20260812 19:13:18.701413 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:18.723098 22008 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:18.723394 21327 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:18.657032 23425 raft_consensus.cc:2781] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:18.571008 23176 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:18.655447 23520 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:18.716367 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:18.595134 21321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1402) took 2287 ms (client timeout 2216 ms). Trace:
I20260812 19:13:18.702133 23014 tablet_bootstrap.cc:492] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
W20260812 19:13:18.726293 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 79770.5us
W20260812 19:13:18.728609 22008 leader_election.cc:334] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.073s (SENT)
I20260812 19:13:18.736387 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:18.740793 22052 raft_consensus.cc:2379] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
W20260812 19:13:18.731642 21321 rpcz_store.cc:269] 0812 19:13:16.285807 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.329039 (+2043232us) service_pool.cc:227] Handling call
0812 19:13:18.573465 (+244426us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:18.742877 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:18.729084 21323 raft_consensus.cc:2396] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260812 19:13:18.726020 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 135702us
I20260812 19:13:18.744822 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1418) took 2336 ms. Trace:
W20260812 19:13:18.726739 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 78211.1us
I20260812 19:13:18.745204 21287 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:18.703145 23385 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:18.738046 21114 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:18.740151 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:18.731407 22056 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:18.762967 21114 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:18.763247 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:18.762988 22053 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:18.748359 21203 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:18.762732 21318 rpcz_store.cc:276] 0812 19:13:16.408874 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.718003 (+2309129us) service_pool.cc:227] Handling call
0812 19:13:18.744103 (+ 26100us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:18.744433 (+ 330us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:18.744461 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:18.744507 (+ 46us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:18.744545 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:18.744605 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:18.744763 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":102}
I20260812 19:13:18.749174 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:18.715145 23435 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:18.748324 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:18.765009 21323 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:18.567044 22046 consensus_queue.cc:260] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:18.770542 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:18.771588 22052 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:18.727325 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 80876us
I20260812 19:13:18.604372 23528 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:18.772339 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:18.725474 23425 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:18.774282 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:18.776258 22051 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:18.774421 21114 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:18.776383 22056 raft_consensus.cc:2379] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:18.768599 21203 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:18.768952 21319 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:18.694247 23364 consensus_queue.cc:1035] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20260812 19:13:18.731169 23551 raft_consensus.cc:2726] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:18.737074 23254 raft_consensus.cc:2781] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:18.731290 23432 raft_consensus.cc:2781] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:18.770560 23255 raft_consensus.cc:2726] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:18.773959 23334 raft_consensus.cc:2726] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:18.800392 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:18.745548 23452 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:13:18.797366 21324 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:18.815501 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:18.746645 23206 tablet_copy_client.cc:791] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 0 data blocks...
I20260812 19:13:18.768676 23385 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:18.819375 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 50946.7us
W20260812 19:13:18.849609 21204 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.70.132:39579 timed out after 1.784s (SENT) [suppressed 30 similar messages]
I20260812 19:13:18.855449 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:18.775082 23515 raft_consensus.cc:2781] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:18.776641 21112 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:18.740126 23520 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:18.725733 23553 consensus_queue.cc:1046] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.164s
I20260812 19:13:18.795208 21323 raft_consensus.cc:1261] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:18.774876 23529 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:19.747852 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 956789us
I20260812 19:13:19.749675 22051 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:19.750955 21329 raft_consensus.cc:1226] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260812 19:13:19.751398 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1685) took 1203 ms. Trace:
I20260812 19:13:19.763440 21324 rpcz_store.cc:276] 0812 19:13:18.547921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.564396 (+ 16475us) service_pool.cc:227] Handling call
0812 19:13:19.751316 (+1186920us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:19.763747 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1428) took 1739 ms. Trace:
I20260812 19:13:19.764143 21323 rpcz_store.cc:276] 0812 19:13:18.024597 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.769196 (+744599us) service_pool.cc:227] Handling call
0812 19:13:18.782822 (+ 13626us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:19.763449 (+980627us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:19.763683 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:19.436236 21114 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.009s sys 0.012s
W20260812 19:13:19.753131 22055 consensus_meta.cc:287] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.095s user 0.021s sys 0.004s
I20260812 19:13:18.889452 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:19.765347 22055 raft_consensus.cc:2454] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:19.765939 22055 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40624 (request call id 360) took 1165 ms. Trace:
W20260812 19:13:19.765275 21114 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.958s user 0.016s sys 0.015s
W20260812 19:13:19.766724 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 959351us
W20260812 19:13:19.755205 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 905593us
I20260812 19:13:18.778435 23459 raft_consensus.cc:2781] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260812 19:13:19.776191 21200 leader_election.cc:334] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.641s (SENT)
I20260812 19:13:19.776443 21114 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:19.765985 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1430) took 1725 ms. Trace:
I20260812 19:13:19.777015 21200 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:19.766255 22055 rpcz_store.cc:276] 0812 19:13:18.600350 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.600612 (+ 262us) service_pool.cc:227] Handling call
0812 19:13:19.765889 (+1165277us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:19.769820 21151 raft_consensus.cc:1226] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 3. Current term is 4. Ops: []
W20260812 19:13:19.761687 22011 connection.cc:232] RPC call timeout handler was delayed by -4.107233286020346s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1642s for a response
I20260812 19:13:19.770524 21203 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:18.776902 23425 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:18.818542 23014 ts_tablet_manager.cc:1397] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 6.222s user 0.076s sys 0.069s
I20260812 19:13:19.775072 21319 raft_consensus.cc:2379] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 4: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:19.778867 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:18.920994 23539 log.cc:924] Time spent T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.160s user 0.000s sys 0.001s
I20260812 19:13:19.777621 21324 rpcz_store.cc:276] 0812 19:13:18.040526 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:19.764362 (+1723836us) service_pool.cc:227] Handling call
0812 19:13:19.764708 (+ 346us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:19.765672 (+ 964us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:19.765699 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:19.765734 (+ 35us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:19.765771 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:19.765829 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:19.765929 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":627}
I20260812 19:13:19.779439 21152 raft_consensus.cc:1226] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260812 19:13:19.791391 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:19.781751 22011 connection.cc:232] RPC call timeout handler was delayed by -4.0370332860620692s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17200000000000001s for a response
W20260812 19:13:19.760281 22052 consensus_meta.cc:287] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.984s user 0.010s sys 0.001s
I20260812 19:13:19.792287 22055 raft_consensus.cc:2354] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3.
I20260812 19:13:18.860783 22046 raft_consensus.cc:3037] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:18.860742 23428 consensus_meta.cc:287] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.888s user 0.006s sys 0.007s
I20260812 19:13:19.779155 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1431) took 1721 ms. Trace:
I20260812 19:13:19.797163 21323 rpcz_store.cc:276] 0812 19:13:18.044317 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:19.764954 (+1720637us) service_pool.cc:227] Handling call
0812 19:13:19.765263 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:19.765437 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:19.765461 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:19.765500 (+ 39us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:19.765528 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:19.765580 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:19.765689 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:19.208359 23470 log.cc:924] Time spent T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 2.842s user 0.004s sys 0.022s
W20260812 19:13:19.457504 23557 log.cc:924] Time spent T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.256s user 0.001s sys 0.001s
W20260812 19:13:19.286180 23550 log.cc:924] Time spent T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.584s user 0.001s sys 0.000s
I20260812 19:13:19.787927 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:19.822036 21158 raft_consensus.cc:2454] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:13:19.784502 22053 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40598 (request call id 577) took 1047 ms. Trace:
I20260812 19:13:19.773546 23548 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:19.040074 23209 consensus_meta.cc:287] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.858s user 0.012s sys 0.000s
W20260812 19:13:19.783176 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 1.00651e+06us
W20260812 19:13:19.792352 22011 connection.cc:232] RPC call timeout handler was delayed by -3.8624332860345021s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19140000000000001s for a response
I20260812 19:13:19.792642 22052 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:19.792846 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1691) took 1003 ms. Trace:
I20260812 19:13:18.864907 23435 raft_consensus.cc:2781] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:19.794644 21287 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:18.890336 23243 raft_consensus.cc:2726] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:19.159926 23400 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:18.912320 23436 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:18.925853 23255 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.893s
I20260812 19:13:18.899808 23432 raft_consensus.cc:686] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:19.819375 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:19.823431 22053 rpcz_store.cc:276] 0812 19:13:18.737759 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.770741 (+ 32982us) service_pool.cc:227] Handling call
0812 19:13:19.784421 (+1013680us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:18.856122 23254 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:19.315800 23371 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:19.782151 23459 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:18.622895 22045 rpcz_store.cc:276] 0812 19:13:16.206808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:16.233268 (+ 26460us) service_pool.cc:227] Handling call
0812 19:13:16.324837 (+ 91569us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:16.347777 (+ 22940us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:16.352169 (+ 4392us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:16.378324 (+ 26155us) log.cc:842] Serialized 26 byte log entry
0812 19:13:18.097298 (+1718974us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:18.103628 (+ 6330us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:18.114354 (+ 10726us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:18.585977 (+471623us) raft_consensus.cc:1671] finished
0812 19:13:18.586004 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:18.586214 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1716333,"threads_started":1,"wal-append.queue_time_us":1287383,"wal-append.run_cpu_time_us":7618,"wal-append.run_wall_time_us":1983448}
I20260812 19:13:19.761933 23515 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:19.793869 22047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 355) took 1690 ms. Trace:
I20260812 19:13:19.774708 23370 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:19.835966 22056 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.675s user 0.008s sys 0.034s
I20260812 19:13:19.826578 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:19.864104 22056 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s user 0.009s sys 0.042s
I20260812 19:13:19.827576 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:19.845532 21318 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.638s user 0.002s sys 0.018s
W20260812 19:13:19.834221 22011 connection.cc:232] RPC call timeout handler was delayed by -3.8129332860698923s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19690000000000002s for a response
I20260812 19:13:19.847074 21112 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:19.834388 21319 rpcz_store.cc:276] 0812 19:13:18.789766 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.810193 (+ 20427us) service_pool.cc:227] Handling call
0812 19:13:19.792781 (+982588us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:19.835578 22048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40598 (request call id 561) took 1120 ms. Trace:
I20260812 19:13:19.350203 23176 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:19.848593 23547 raft_consensus.cc:2726] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:13:19.853627 21323 raft_consensus.cc:1226] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:19.855099 23254 raft_consensus.cc:3037] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:19.823917 23428 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:19.850768 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:19.856411 21158 raft_consensus.cc:1226] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:19.825143 23549 raft_consensus.cc:2781] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260812 19:13:19.861241 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 66768.5us
I20260812 19:13:19.837689 23206 tablet_copy_client.cc:655] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Starting download of 1 WAL segments...
I20260812 19:13:19.865000 22055 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:19.834455 22052 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40598 (request call id 574) took 1115 ms. Trace:
I20260812 19:13:19.875566 23490 consensus_queue.cc:1046] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.010s
I20260812 19:13:19.882565 22052 rpcz_store.cc:276] 0812 19:13:18.719123 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.770233 (+ 51110us) service_pool.cc:227] Handling call
0812 19:13:19.834383 (+1064150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:19.862854 23371 raft_consensus.cc:2726] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:19.884536 21287 leader_election.cc:302] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
W20260812 19:13:19.865011 21318 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s user 0.002s sys 0.018s
I20260812 19:13:19.848861 23452 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:19.851917 21320 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.032s user 0.035s sys 0.031s
I20260812 19:13:19.856489 23459 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:19.854346 23435 raft_consensus.cc:686] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:19.873252 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1680) took 1342 ms. Trace:
I20260812 19:13:19.874204 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:19.858170 23243 raft_consensus.cc:2726] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:13:19.817798 23553 raft_consensus.cc:2781] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:19.875741 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1665) took 5940 ms. Trace:
I20260812 19:13:19.861804 23515 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:19.851135 23209 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:19.879011 23432 consensus_queue.cc:237] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:19.846393 23548 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:19.887976 21320 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.068s user 0.035s sys 0.031s
W20260812 19:13:19.879988 23561 log.cc:924] Time spent T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.092s user 0.000s sys 0.001s
I20260812 19:13:19.879835 23529 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:19.890527 21323 rpcz_store.cc:276] 0812 19:13:18.531054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:19.853025 (+1321971us) service_pool.cc:227] Handling call
0812 19:13:19.853383 (+ 358us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:19.872999 (+ 19616us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:19.873201 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:19.892288 23547 raft_consensus.cc:2741] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260812 19:13:19.895141 21328 rpcz_store.cc:276] 0812 19:13:13.931799 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:14.936241 (+1004442us) service_pool.cc:227] Handling call
0812 19:13:14.936733 (+ 492us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:14.937177 (+ 444us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:14.937218 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:14.944315 (+ 7097us) log.cc:842] Serialized 26 byte log entry
0812 19:13:15.975169 (+1030854us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:15.975223 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:15.975331 (+ 108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:18.735058 (+2759727us) spinlock_profiling.cc:243] Waited 4.86 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:19.866262 (+1131204us) raft_consensus.cc:1671] finished
0812 19:13:19.866277 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:19.872370 (+ 6093us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":45776,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":131,"spinlock_wait_cycles":11183360,"thread_start_us":697856,"threads_started":1,"wal-append.queue_time_us":1146639}
I20260812 19:13:19.907336 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:19.908377 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1683) took 1361 ms. Trace:
I20260812 19:13:19.908732 21328 rpcz_store.cc:276] 0812 19:13:18.547109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:19.907581 (+1360472us) service_pool.cc:227] Handling call
0812 19:13:19.907897 (+ 316us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:19.908098 (+ 201us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:19.908115 (+ 17us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:19.908142 (+ 27us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:19.908163 (+ 21us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:19.908201 (+ 38us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:19.908293 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:19.882043 23549 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:19.901859 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:19.868165 22048 rpcz_store.cc:276] 0812 19:13:18.705311 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.721420 (+ 16109us) service_pool.cc:227] Handling call
0812 19:13:19.825539 (+1104119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:19.912074 22056 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 567) took 1199 ms. Trace:
I20260812 19:13:19.900476 23255 raft_consensus.cc:2781] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:19.916222 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:19.919116 21319 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:19.899583 23553 raft_consensus.cc:686] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:19.914731 22056 rpcz_store.cc:276] 0812 19:13:18.713032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.802772 (+ 89740us) service_pool.cc:227] Handling call
0812 19:13:18.803363 (+ 591us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:18.817029 (+ 13666us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:18.817062 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:18.863069 (+ 46007us) log.cc:842] Serialized 26 byte log entry
0812 19:13:19.864480 (+1001411us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:19.864509 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:19.864568 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:19.911798 (+ 47230us) raft_consensus.cc:1671] finished
0812 19:13:19.911820 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:19.911990 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13103,"raft.queue_time_us":55410,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":60403,"thread_start_us":703060,"threads_started":1,"wal-append.queue_time_us":950419}
I20260812 19:13:19.928349 21287 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:19.862010 22047 rpcz_store.cc:276] 0812 19:13:18.097699 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.100993 (+ 3294us) service_pool.cc:227] Handling call
0812 19:13:18.156015 (+ 55022us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:18.374365 (+218350us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:18.374402 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:18.375011 (+ 609us) log.cc:842] Serialized 26 byte log entry
0812 19:13:18.564100 (+189089us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:18.568072 (+ 3972us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:19.473504 (+905432us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b5400430654. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:19.480679 (+ 7175us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:19.485723 (+ 5044us) raft_consensus.cc:1671] finished
0812 19:13:19.510518 (+ 24795us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:19.787840 (+277322us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1920,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"spinlock_wait_cycles":465857792,"thread_start_us":163527,"threads_started":1,"wal-append.queue_time_us":201491,"wal-append.run_cpu_time_us":1981,"wal-append.run_wall_time_us":1073270}
I20260812 19:13:19.931275 23209 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:19.931708 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1411) took 4552 ms. Trace:
W20260812 19:13:19.929937 21153 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1678) took 4621 ms (client timeout 1925 ms). Trace:
W20260812 19:13:19.933610 21153 rpcz_store.cc:269] 0812 19:13:15.306037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.308700 (+ 2663us) service_pool.cc:227] Handling call
0812 19:13:19.927188 (+4618488us) spinlock_profiling.cc:243] Waited 2.87 s on lock 0x7b54001955d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:19.927368 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6608655744}
I20260812 19:13:19.932147 21239 rpcz_store.cc:276] 0812 19:13:15.379329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:15.386976 (+ 7647us) service_pool.cc:227] Handling call
0812 19:13:15.387349 (+ 373us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:18.481741 (+3094392us) raft_consensus.cc:1520] Early marking committed up to index 2
0812 19:13:18.481786 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:18.481851 (+ 65us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:18.481904 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:19.931368 (+1449464us) spinlock_profiling.cc:243] Waited 2.69 s on lock 0x7b5400303754. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:19.931420 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:19.931607 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6647,"raft.queue_time_us":39178,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":42367,"spinlock_wait_cycles":6178992384,"thread_start_us":324495,"threads_started":1}
I20260812 19:13:19.893119 23452 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:19.937059 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:19.912789 23549 raft_consensus.cc:3037] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
W20260812 19:13:19.933887 23562 log.cc:924] Time spent T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20260812 19:13:19.941854 23528 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:19.918705 23255 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:19.899209 23435 consensus_queue.cc:237] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:19.967207 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1425) took 1975 ms. Trace:
I20260812 19:13:19.967887 21321 rpcz_store.cc:276] 0812 19:13:17.992294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.742721 (+750427us) service_pool.cc:227] Handling call
0812 19:13:18.743035 (+ 314us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:18.764383 (+ 21348us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:18.764416 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:18.764901 (+ 485us) log.cc:842] Serialized 26 byte log entry
0812 19:13:18.923171 (+158270us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:18.923232 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:18.923352 (+ 120us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:19.966879 (+1043527us) raft_consensus.cc:1671] finished
0812 19:13:19.966901 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:19.967114 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":186,"spinlock_wait_cycles":29290880,"thread_start_us":119915,"threads_started":1,"wal-append.queue_time_us":1084392}
W20260812 19:13:19.978093 21327 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.761s user 0.002s sys 0.007s
I20260812 19:13:19.959098 23255 raft_consensus.cc:3037] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:19.978672 21327 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.762s user 0.002s sys 0.007s
I20260812 19:13:19.980991 23547 consensus_queue.cc:1046] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:13:19.983043 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:19.986093 21319 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:19.983402 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:19.992736 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:19.997502 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:19.998020 21323 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:19.998232 21319 raft_consensus.cc:2362] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4. Re-sending same reply.
I20260812 19:13:19.999264 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:19.999361 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:20.000465 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:19.965099 23014 raft_consensus.cc:348] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:20.002839 23014 raft_consensus.cc:729] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:13:20.003710 23014 consensus_queue.cc:260] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
W20260812 19:13:19.978968 23566 log.cc:924] Time spent T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20260812 19:13:19.996476 21321 raft_consensus.cc:1261] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260812 19:13:20.009016 21319 raft_consensus.cc:2379] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:20.005278 22051 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:20.013401 22051 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40598 (request call id 568) took 1300 ms. Trace:
I20260812 19:13:20.014046 22051 rpcz_store.cc:276] 0812 19:13:18.713342 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.798954 (+ 85612us) service_pool.cc:227] Handling call
0812 19:13:20.013316 (+1214362us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:20.020035 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1420) took 3576 ms. Trace:
I20260812 19:13:20.020599 21327 rpcz_store.cc:276] 0812 19:13:16.443911 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.724213 (+2280302us) service_pool.cc:227] Handling call
0812 19:13:18.747357 (+ 23144us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:18.799265 (+ 51908us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:18.799305 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:18.892595 (+ 93290us) log.cc:842] Serialized 26 byte log entry
0812 19:13:19.985779 (+1093184us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:19.985822 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:19.985916 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:20.019755 (+ 33839us) raft_consensus.cc:1671] finished
0812 19:13:20.019786 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:20.019970 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":203336,"raft.queue_time_us":51877,"raft.run_cpu_time_us":380,"raft.run_wall_time_us":93204,"thread_start_us":768508,"threads_started":1,"wal-append.queue_time_us":970152}
W20260812 19:13:20.024973 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 248641us
I20260812 19:13:20.026036 21114 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:20.027761 21114 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:19.984301 23452 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:19.995828 23287 raft_consensus.cc:2726] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:20.001559 23548 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:20.089366 22053 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 569) took 1375 ms. Trace:
I20260812 19:13:20.079717 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1423) took 2111 ms. Trace:
W20260812 19:13:20.074102 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 46459.3us
I20260812 19:13:20.097728 22053 rpcz_store.cc:276] 0812 19:13:18.713645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:19.856829 (+1143184us) service_pool.cc:227] Handling call
0812 19:13:19.868771 (+ 11942us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:19.877528 (+ 8757us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:19.877551 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:19.877914 (+ 363us) log.cc:842] Serialized 26 byte log entry
0812 19:13:20.030358 (+152444us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:20.030409 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:20.030502 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:20.089026 (+ 58524us) raft_consensus.cc:1671] finished
0812 19:13:20.089047 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:20.089289 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":220514,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":187,"thread_start_us":135415,"threads_started":1,"wal-append.queue_time_us":193395}
I20260812 19:13:20.099449 21114 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:20.091846 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 226207us
I20260812 19:13:20.100594 22011 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:20.092468 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 224594us
I20260812 19:13:20.103291 21112 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:20.104372 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1679) took 1573 ms. Trace:
I20260812 19:13:20.104630 22011 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:20.014137 23428 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:20.097867 21320 rpcz_store.cc:276] 0812 19:13:17.968399 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.729297 (+760898us) service_pool.cc:227] Handling call
0812 19:13:18.729577 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:18.729760 (+ 183us) raft_consensus.cc:1520] Early marking committed up to index 2
0812 19:13:18.820082 (+ 90322us) log.cc:842] Serialized 14 byte log entry
0812 19:13:20.079151 (+1259069us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:20.079235 (+ 84us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:20.079318 (+ 83us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:20.079422 (+ 104us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:20.079622 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1065625,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":45,"raft.queue_time_us":5283,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":4296,"thread_start_us":1330718,"threads_started":3,"wal-append.queue_time_us":1070946}
I20260812 19:13:20.088702 23243 raft_consensus.cc:2781] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:20.110126 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:20.111204 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:20.106408 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 330230us
I20260812 19:13:20.066776 23584 raft_consensus.cc:2726] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:20.114344 21114 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:20.116628 21112 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:20.118191 21320 raft_consensus.cc:1261] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260812 19:13:20.118156 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:20.104820 21324 rpcz_store.cc:276] 0812 19:13:18.530636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:19.791951 (+1261315us) service_pool.cc:227] Handling call
0812 19:13:19.836436 (+ 44485us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:19.836737 (+ 301us) raft_consensus.cc:1520] Early marking committed up to index 6
0812 19:13:19.854724 (+ 17987us) log.cc:842] Serialized 14 byte log entry
0812 19:13:20.104007 (+249283us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:20.104063 (+ 56us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:20.104110 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:20.104176 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:20.104307 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1708,"prepare.queue_time_us":19084,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":3372,"raft.queue_time_us":76788,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":9473,"thread_start_us":246523,"threads_started":2,"wal-append.queue_time_us":163410}
I20260812 19:13:20.119796 22055 raft_consensus.cc:2379] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 2: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:20.121742 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:20.113144 23583 raft_consensus.cc:2726] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:19.983045 23553 consensus_queue.cc:237] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:20.124822 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:20.129242 22053 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:20.123898 21320 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:20.025302 23515 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:20.126263 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:20.130697 21319 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:20.122283 21204 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:20.128254 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 357775us
I20260812 19:13:20.132447 21203 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:20.134677 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:20.136230 23515 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:20.136410 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:20.109647 23576 raft_consensus.cc:2726] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:20.114957 23243 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:20.144675 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:20.145300 21324 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:20.148507 21114 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:20.123616 23435 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:20.148895 23584 raft_consensus.cc:2726] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:20.154070 22048 raft_consensus.cc:1226] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:20.127900 22045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 571) took 1412 ms. Trace:
I20260812 19:13:20.132627 23573 raft_consensus.cc:2726] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:20.156587 23572 raft_consensus.cc:2781] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:20.156972 23243 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:20.158958 23014 ts_tablet_manager.cc:1428] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.354s user 0.007s sys 0.023s
I20260812 19:13:20.160861 21200 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:20.175032 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 58504us
I20260812 19:13:20.172099 21319 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
W20260812 19:13:20.181543 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 59343.8us
I20260812 19:13:20.182649 21204 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:20.184414 21204 leader_election.cc:398] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Vote denied by peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579) with higher term. Message: Invalid argument: T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3.
I20260812 19:13:20.184799 21204 leader_election.cc:401] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:13:20.149413 23565 raft_consensus.cc:2781] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:20.190505 21200 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:20.158995 23515 raft_consensus.cc:2781] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:13:20.191126 21112 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:20.191318 21204 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:20.186129 23584 raft_consensus.cc:2726] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:20.187513 23576 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:20.204703 21324 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
W20260812 19:13:20.206192 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 53967.7us
I20260812 19:13:20.180210 23572 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:20.191677 23371 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 3
I20260812 19:13:20.220794 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:20.223407 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:20.224375 21239 raft_consensus.cc:3037] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:20.224437 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:20.224871 21115 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:20.225546 21244 raft_consensus.cc:2379] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 3: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:20.223675 21243 raft_consensus.cc:1226] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:20.218170 23572 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:20.227814 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:20.228485 21241 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:20.232544 21239 raft_consensus.cc:2454] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:13:20.246387 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:20.246696 23601 consensus_queue.cc:1046] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260812 19:13:20.177167 23435 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:20.249415 23435 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:20.251021 23477 consensus_meta.cc:287] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 1.593s user 0.012s sys 0.005s
I20260812 19:13:20.251606 23477 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:20.254127 23477 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:20.196970 23370 raft_consensus.cc:2781] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:20.254891 23370 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:20.255285 23370 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:20.259074 23429 raft_consensus.cc:2781] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:20.259593 23429 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:20.259929 23429 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:20.194070 23575 raft_consensus.cc:2781] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:20.263027 23575 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:20.263427 23575 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:20.191890 23400 raft_consensus.cc:2726] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:20.265695 23599 raft_consensus.cc:2726] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:20.267145 21328 raft_consensus.cc:1226] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: []
W20260812 19:13:20.276333 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 57576.9us
W20260812 19:13:20.276482 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 51696.1us
I20260812 19:13:20.276953 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:20.150380 23583 consensus_queue.cc:1046] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260812 19:13:20.290230 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:20.290369 23583 raft_consensus.cc:3032] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:20.291160 23583 raft_consensus.cc:729] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:20.196470 23565 raft_consensus.cc:686] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:20.198045 23428 consensus_queue.cc:1046] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.097s
W20260812 19:13:20.310664 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 120240us
I20260812 19:13:20.300251 23601 raft_consensus.cc:3032] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:20.299824 23583 consensus_queue.cc:260] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:20.301071 21115 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:20.303539 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:20.315277 21241 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:20.240778 23371 raft_consensus.cc:2726] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579) with higher term. Message: Invalid argument: T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3.
W20260812 19:13:20.319082 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 125278us
I20260812 19:13:20.266836 23574 raft_consensus.cc:2781] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:20.293416 23584 raft_consensus.cc:2781] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:20.314551 23601 raft_consensus.cc:729] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:20.337890 23176 consensus_queue.cc:1046] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:13:20.338195 23583 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:20.341908 21200 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:20.338932 23584 raft_consensus.cc:686] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:13:20.322032 22046 consensus_meta.cc:287] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.525s user 0.008s sys 0.004s
W20260812 19:13:20.352921 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 161870us
I20260812 19:13:20.354238 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.359853 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.366873 21200 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:20.377669 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:20.362381 22046 raft_consensus.cc:1261] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260812 19:13:20.382480 22046 pending_rounds.cc:76] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706: Aborting all ops after (but not including) 5
I20260812 19:13:20.382830 22046 pending_rounds.cc:97] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260812 19:13:20.382589 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.383177 22046 raft_consensus.cc:2866] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:20.383471 23371 raft_consensus.cc:2781] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:20.383961 23371 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:20.384126 22046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 354) took 2332 ms. Trace:
I20260812 19:13:20.384490 23371 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:20.384629 22046 rpcz_store.cc:276] 0812 19:13:18.052092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.073491 (+ 21399us) service_pool.cc:227] Handling call
0812 19:13:18.083206 (+ 9715us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:20.383783 (+2300577us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:20.384061 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:20.382584 21200 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:20.359786 23573 raft_consensus.cc:2781] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:20.387079 23573 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:20.387467 23573 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:20.388275 21200 leader_election.cc:302] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:20.384009 23243 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:20.338553 23601 consensus_queue.cc:260] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:20.389618 21200 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:20.391157 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:20.391305 23455 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:20.362346 23584 consensus_queue.cc:237] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:20.347220 23515 raft_consensus.cc:686] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:20.344897 23422 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:20.398391 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.399089 21237 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
W20260812 19:13:20.403499 21204 leader_election.cc:334] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Timed out: RequestConsensusVote RPC to 127.19.70.132:39579 timed out after 1.573s (SENT)
W20260812 19:13:20.405113 21204 leader_election.cc:334] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Timed out: RequestConsensusVote RPC to 127.19.70.132:39579 timed out after 1.578s (SENT)
I20260812 19:13:20.406487 21157 raft_consensus.cc:2379] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:20.350036 23574 raft_consensus.cc:686] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:20.282178 23576 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:20.414832 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.410436 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:20.395712 23243 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:20.423504 23243 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:20.392798 23612 raft_consensus.cc:2781] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:20.427310 23612 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:20.427953 23612 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:20.418319 23254 consensus_meta.cc:287] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.537s user 0.017s sys 0.002s
I20260812 19:13:20.431275 23254 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:20.432960 21153 raft_consensus.cc:2354] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3.
I20260812 19:13:20.432523 23549 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:20.435293 23515 consensus_queue.cc:237] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:20.434190 23254 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:20.414978 23425 consensus_meta.cc:287] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.618s user 0.007s sys 0.009s
I20260812 19:13:20.445981 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.407635 23601 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:20.446887 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.413784 23422 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:20.176227 22045 rpcz_store.cc:276] 0812 19:13:18.714188 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:19.868900 (+1154712us) service_pool.cc:227] Handling call
0812 19:13:19.908479 (+ 39579us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:20.096390 (+187911us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:20.096426 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:20.096771 (+ 345us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:20.107455 (+ 10684us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:20.114530 (+ 7075us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:20.126172 (+ 11642us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":160843,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":75,"thread_start_us":158444,"threads_started":1}
I20260812 19:13:20.410544 23609 raft_consensus.cc:2781] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:20.395172 23455 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:20.459558 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 156838us
I20260812 19:13:20.470970 21151 raft_consensus.cc:1226] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260812 19:13:20.443938 23528 raft_consensus.cc:2726] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:20.474488 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:20.473567 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 112794us
I20260812 19:13:20.477878 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:20.478163 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:20.478727 21112 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:20.481083 21241 raft_consensus.cc:1261] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260812 19:13:20.483768 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:20.484869 21241 raft_consensus.cc:2379] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:20.489024 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:20.491066 21233 raft_consensus.cc:1261] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260812 19:13:20.492197 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:20.493103 21244 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
W20260812 19:13:20.500905 23459 consensus_meta.cc:287] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.608s user 0.007s sys 0.008s
I20260812 19:13:20.501605 23459 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:20.461964 23425 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:20.502849 23572 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:20.504046 23372 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:20.505249 23549 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:20.505331 23425 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:20.450824 23255 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:20.506150 23425 raft_consensus.cc:2726] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:20.472332 23611 raft_consensus.cc:2726] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:20.507591 21157 raft_consensus.cc:1261] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260812 19:13:20.457829 23520 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:20.508389 23422 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:20.520349 23255 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:20.520618 23502 raft_consensus.cc:1050] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
W20260812 19:13:20.521186 23502 raft_consensus.cc:1054] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: Unable to promote non-voter 63db10063a7d4eb2a9cc9dc952654706: Illegal state: Leader has not yet committed an operation in its own term
I20260812 19:13:20.522747 23429 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:20.457883 23576 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:20.423520 23603 consensus_queue.cc:1035] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.085s
I20260812 19:13:20.509743 23617 consensus_queue.cc:1035] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 19:13:20.437880 23565 consensus_queue.cc:237] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:20.516971 23459 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:20.448808 23610 consensus_queue.cc:1035] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260812 19:13:20.538771 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.551700 21157 raft_consensus.cc:1226] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:20.559751 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.558365 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:13:20.530180 21114 connection.cc:232] RPC call timeout handler was delayed by -2.68373170401901s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2137s for a response
I20260812 19:13:20.580288 21240 raft_consensus.cc:1226] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260812 19:13:20.573998 21114 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:20.587383 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:20.589919 23459 raft_consensus.cc:2726] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:20.590813 21155 raft_consensus.cc:2379] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:20.591056 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:20.591991 21236 raft_consensus.cc:2379] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:20.592981 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:20.594687 21244 raft_consensus.cc:1226] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260812 19:13:20.595837 23511 raft_consensus.cc:3032] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:20.596078 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:20.596460 23511 raft_consensus.cc:729] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:20.594285 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:20.598212 23393 raft_consensus.cc:3032] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:20.598492 21236 raft_consensus.cc:3037] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:20.599093 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:20.601063 21244 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:20.598734 23393 raft_consensus.cc:729] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:20.603652 23393 consensus_queue.cc:260] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:20.604401 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.604775 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.605023 23393 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:20.605742 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:20.607908 21203 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:20.610168 21203 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:20.612102 21203 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:20.613350 21114 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:20.621730 21203 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:20.600360 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:20.581609 23385 consensus_meta.cc:287] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.797s user 0.016s sys 0.000s
I20260812 19:13:20.627701 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
W20260812 19:13:20.628093 21287 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.016s sys 0.027s
I20260812 19:13:20.628229 21154 raft_consensus.cc:1226] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: [4.4-4.4]
W20260812 19:13:20.631125 21287 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.597s user 0.016s sys 0.027s
W20260812 19:13:20.631440 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 597639us
I20260812 19:13:20.632261 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:20.633177 21155 raft_consensus.cc:2379] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:20.633636 23619 raft_consensus.cc:1050] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
W20260812 19:13:20.634088 23619 raft_consensus.cc:1054] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: Unable to promote non-voter 63db10063a7d4eb2a9cc9dc952654706: Illegal state: Leader has not yet committed an operation in its own term
I20260812 19:13:20.628471 23432 consensus_queue.cc:1035] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260812 19:13:20.558882 23574 consensus_queue.cc:237] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:20.570057 23176 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:20.530768 23609 raft_consensus.cc:686] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:20.633431 21287 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
W20260812 19:13:20.658941 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 46815.7us
I20260812 19:13:20.670802 21236 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:20.626019 23611 raft_consensus.cc:2726] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:20.624492 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:20.623777 23429 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:20.597836 23511 consensus_queue.cc:260] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:20.630577 23528 raft_consensus.cc:2726] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:13:20.663132 23422 raft_consensus.cc:2726] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:20.678704 22050 raft_consensus.cc:1261] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260812 19:13:20.679159 23572 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:20.665542 23425 consensus_queue.cc:1035] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260812 19:13:20.694175 21244 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:20.668957 23432 raft_consensus.cc:2726] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:20.701021 21114 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:20.703289 22050 pending_rounds.cc:76] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706: Aborting all ops after (but not including) 4
I20260812 19:13:20.656320 23385 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:20.709762 23313 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:20.678726 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:20.744700 21114 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:20.735160 22011 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:20.705974 23597 consensus_queue.cc:1035] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260812 19:13:20.738013 22050 pending_rounds.cc:97] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260812 19:13:20.714112 23623 consensus_queue.cc:1035] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260812 19:13:20.667832 23176 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:20.745069 21287 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:20.746556 23583 raft_consensus.cc:2781] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260812 19:13:20.750697 21114 leader_election.cc:334] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 2.137s (SENT)
I20260812 19:13:20.708526 23594 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:20.714937 23511 raft_consensus.cc:3037] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:20.763871 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.766361 21152 raft_consensus.cc:1226] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:20.766582 21154 raft_consensus.cc:2379] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:20.773077 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:20.772845 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 292733us
I20260812 19:13:20.746002 23432 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:20.748188 22050 raft_consensus.cc:2866] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260812 19:13:20.766345 23370 consensus_meta.cc:287] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.511s user 0.010s sys 0.002s
I20260812 19:13:20.777379 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.758159 22047 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:20.779408 21151 raft_consensus.cc:2354] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5.
I20260812 19:13:20.766755 23422 raft_consensus.cc:2781] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:20.783541 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.784060 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.757272 23605 raft_consensus.cc:2781] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:20.785106 21156 raft_consensus.cc:2396] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260812 19:13:20.753705 23583 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:20.788812 21204 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:20.771003 23520 raft_consensus.cc:2726] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:20.792068 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.793324 21154 raft_consensus.cc:2379] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:20.791579 23528 raft_consensus.cc:2726] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:20.777252 23432 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:20.794796 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.754256 23572 consensus_queue.cc:1046] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260812 19:13:20.796837 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.809890 21156 raft_consensus.cc:2354] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4.
I20260812 19:13:20.800201 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:20.811538 21321 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:20.812608 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:20.757782 23603 consensus_queue.cc:1046] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260812 19:13:20.803041 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:20.802172 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:20.822059 22011 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:20.799151 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:20.816295 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:20.824429 22011 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:20.824494 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:20.822978 21324 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:20.826712 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:20.827530 21324 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:20.801162 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:20.828382 21320 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:20.830013 21323 raft_consensus.cc:2379] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:20.793910 23583 raft_consensus.cc:3037] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:20.831665 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:20.792346 23372 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:20.807510 23206 tablet_copy_client.cc:531] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 19:13:20.757534 23313 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:20.839000 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:20.856658 21320 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:20.837365 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:20.848899 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:20.769275 23615 consensus_queue.cc:1046] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.037s
I20260812 19:13:20.742089 23609 consensus_queue.cc:237] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:20.854324 21319 raft_consensus.cc:1261] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260812 19:13:20.857851 21323 raft_consensus.cc:3037] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:20.813966 23605 raft_consensus.cc:686] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:20.781409 23370 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:20.770766 23594 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:20.813108 23432 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:22.541296 21115 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.714s user 0.010s sys 0.007s
I20260812 19:13:20.981575 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:22.542054 21115 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.731s user 0.010s sys 0.007s
W20260812 19:13:21.451924 23583 consensus_meta.cc:287] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.590s user 0.010s sys 0.002s
I20260812 19:13:20.884371 23584 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:22.546563 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1416) took 6150 ms. Trace:
W20260812 19:13:22.547000 21324 consensus_meta.cc:287] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 1.706s user 0.009s sys 0.003s
W20260812 19:13:22.547833 22053 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.692s user 0.007s sys 0.004s
W20260812 19:13:22.548419 22053 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.841s user 0.008s sys 0.004s
W20260812 19:13:22.382033 21112 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.568s user 0.002s sys 0.000s
I20260812 19:13:22.547861 23584 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
W20260812 19:13:22.550063 21112 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.739s user 0.011s sys 0.002s
I20260812 19:13:22.550238 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1443) took 1751 ms. Trace:
W20260812 19:13:21.559006 23596 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.020s user 0.033s sys 0.083s
W20260812 19:13:21.561661 23603 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.000s sys 0.002s
W20260812 19:13:21.561467 23565 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.729s user 0.000s sys 0.001s
W20260812 19:13:21.992919 23572 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.136s user 0.002s sys 0.009s
W20260812 19:13:21.560616 23515 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.953s user 0.009s sys 0.047s
W20260812 19:13:21.543109 23511 consensus_meta.cc:287] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.774s user 0.013s sys 0.000s
W20260812 19:13:22.524117 21236 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.792s user 0.005s sys 0.014s
W20260812 19:13:20.879750 23575 consensus_meta.cc:287] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.616s user 0.017s sys 0.004s
W20260812 19:13:21.527715 23612 consensus_meta.cc:287] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.099s user 0.017s sys 0.000s
W20260812 19:13:22.534157 22010 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.658s user 0.017s sys 0.020s
W20260812 19:13:21.584591 23574 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.945s user 0.007s sys 0.047s
W20260812 19:13:22.546828 22056 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.671s user 0.002s sys 0.008s
I20260812 19:13:22.547612 21324 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
W20260812 19:13:20.951009 21200 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.029s sys 0.039s
W20260812 19:13:22.382342 22055 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.805s user 0.018s sys 0.030s
I20260812 19:13:22.548105 21327 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 5
W20260812 19:13:21.014525 23371 consensus_meta.cc:287] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.629s user 0.013s sys 0.001s
W20260812 19:13:21.992671 23601 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.884s user 0.000s sys 0.010s
W20260812 19:13:21.832214 23576 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.808s user 0.005s sys 0.004s
W20260812 19:13:22.533114 21239 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.819s user 0.013s sys 0.002s
W20260812 19:13:22.567939 22010 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.720s user 0.018s sys 0.020s
W20260812 19:13:21.023715 23573 consensus_meta.cc:287] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.636s user 0.015s sys 0.000s
I20260812 19:13:22.550731 21319 rpcz_store.cc:276] 0812 19:13:20.803879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.853438 (+ 49559us) service_pool.cc:227] Handling call
0812 19:13:20.853844 (+ 406us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:22.549522 (+1695678us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:22.550047 (+ 525us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:22.547246 21329 rpcz_store.cc:276] 0812 19:13:16.395593 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.704548 (+2308955us) service_pool.cc:227] Handling call
0812 19:13:19.750701 (+1046153us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:19.773573 (+ 22872us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:22.545837 (+2772264us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540023dcd0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:22.546250 (+ 413us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2354666240}
I20260812 19:13:22.545032 23583 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:20.945369 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 323744us
I20260812 19:13:21.501186 23451 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:22.560525 21113 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.658s user 0.023s sys 0.019s
W20260812 19:13:22.567004 23596 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.086s user 0.065s sys 0.131s
W20260812 19:13:22.576653 23603 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.701s user 0.005s sys 0.016s
I20260812 19:13:21.022503 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:22.576488 23572 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.716s user 0.006s sys 0.013s
W20260812 19:13:22.570276 23422 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.718s user 0.000s sys 0.018s
W20260812 19:13:22.575585 23443 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.271s user 0.002s sys 0.013s
W20260812 19:13:22.578887 21236 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.847s user 0.005s sys 0.014s
W20260812 19:13:22.567401 21241 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.760s user 0.005s sys 0.011s
W20260812 19:13:22.572824 22055 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.996s user 0.025s sys 0.038s
W20260812 19:13:22.576134 22011 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.691s user 0.000s sys 0.010s
I20260812 19:13:20.881120 23313 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:22.580789 23255 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.665s user 0.001s sys 0.009s
W20260812 19:13:22.582723 21113 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.733s user 0.024s sys 0.020s
I20260812 19:13:21.235206 23590 raft_consensus.cc:3032] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Stepping down as leader of term 1
W20260812 19:13:22.583727 22056 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.855s user 0.003s sys 0.008s
W20260812 19:13:22.583864 21239 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.916s user 0.013s sys 0.003s
I20260812 19:13:21.064198 23592 raft_consensus.cc:2781] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election won for term 2
W20260812 19:13:22.582871 21200 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.192s user 0.049s sys 0.059s
W20260812 19:13:22.599097 22011 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.774s user 0.000s sys 0.011s
W20260812 19:13:22.611342 21241 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.868s user 0.006s sys 0.011s
I20260812 19:13:22.613402 23451 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:22.614300 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1686) took 4003 ms. Trace:
I20260812 19:13:22.613763 23371 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:22.614763 21328 rpcz_store.cc:276] 0812 19:13:18.568358 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:19.909500 (+1341142us) service_pool.cc:227] Handling call
0812 19:13:19.909868 (+ 368us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:20.282959 (+373091us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:22.508342 (+2225383us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b540004f1d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:22.571592 (+ 63250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":821171584}
I20260812 19:13:22.617086 23451 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:22.617167 23565 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.740s user 0.010s sys 0.010s
I20260812 19:13:22.617663 23642 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:22.618108 23371 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:22.611420 23255 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.760s user 0.001s sys 0.009s
I20260812 19:13:22.618870 23642 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:22.621466 23584 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:22.622349 21324 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1448) took 1791 ms (client timeout 2247 ms). Trace:
W20260812 19:13:22.625330 21324 rpcz_store.cc:269] 0812 19:13:20.805782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.826348 (+ 20566us) service_pool.cc:227] Handling call
0812 19:13:22.584434 (+1758086us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:22.626238 21288 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.674s user 0.008s sys 0.049s
W20260812 19:13:22.626986 21288 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.769s user 0.008s sys 0.051s
I20260812 19:13:22.627537 21240 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 6: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
W20260812 19:13:22.627779 23544 log.cc:924] Time spent T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 3.839s user 0.006s sys 0.022s
I20260812 19:13:20.879674 23176 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:22.629037 23573 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:21.012403 23385 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:20.856298 23372 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:22.634545 21203 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
W20260812 19:13:22.625352 23601 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.494s user 0.001s sys 0.018s
I20260812 19:13:22.624755 21329 raft_consensus.cc:1261] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:13:22.632315 23573 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:22.643535 23452 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.503s user 0.005s sys 0.032s
I20260812 19:13:22.626374 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1448) took 2076 ms. Trace:
W20260812 19:13:22.622449 23443 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.363s user 0.002s sys 0.014s
I20260812 19:13:22.660318 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1452) took 1772 ms. Trace:
W20260812 19:13:22.623847 23422 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.811s user 0.002s sys 0.018s
W20260812 19:13:22.659870 23634 log.cc:924] Time spent T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 1.335s user 0.006s sys 0.008s
I20260812 19:13:22.675839 21329 rpcz_store.cc:276] 0812 19:13:20.887458 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:22.620949 (+1733491us) service_pool.cc:227] Handling call
0812 19:13:22.621634 (+ 685us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:22.644788 (+ 23154us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:22.660209 (+ 15421us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:22.660503 21240 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1458) took 2053 ms (client timeout 2413 ms). Trace:
I20260812 19:13:22.661149 21243 rpcz_store.cc:276] 0812 19:13:20.487841 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.487981 (+ 140us) service_pool.cc:227] Handling call
0812 19:13:20.488414 (+ 433us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:20.488868 (+ 454us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:20.488894 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:20.489373 (+ 479us) log.cc:842] Serialized 26 byte log entry
0812 19:13:20.768422 (+279049us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:20.768458 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:20.768526 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:22.480034 (+1711508us) raft_consensus.cc:1671] finished
0812 19:13:22.497896 (+ 17862us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:22.564194 (+ 66298us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":112,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":243,"thread_start_us":278619,"threads_started":1,"wal-append.queue_time_us":1065164}
I20260812 19:13:22.697270 21321 raft_consensus.cc:3037] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 3
I20260812 19:13:22.702971 23176 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:22.703971 23176 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:22.704636 23380 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:22.689234 23451 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:22.705560 23594 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:22.705853 23451 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:22.707151 21233 raft_consensus.cc:2379] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:22.707401 21236 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1456) took 2141 ms. Trace:
I20260812 19:13:22.708729 23451 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:22.707849 21236 rpcz_store.cc:276] 0812 19:13:20.565890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.672607 (+106717us) service_pool.cc:227] Handling call
0812 19:13:20.699522 (+ 26915us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:20.699784 (+ 262us) raft_consensus.cc:1520] Early marking committed up to index 2
0812 19:13:20.732052 (+ 32268us) log.cc:842] Serialized 14 byte log entry
0812 19:13:22.706826 (+1974774us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:22.706949 (+ 123us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:22.707054 (+ 105us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:22.707165 (+ 111us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:22.707332 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62772,"prepare.queue_time_us":32065,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"thread_start_us":1867806,"threads_started":1}
I20260812 19:13:22.710100 23451 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:22.703246 21320 consensus_meta.cc:287] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 1.692s user 0.054s sys 0.083s
W20260812 19:13:22.685317 21240 rpcz_store.cc:269] 0812 19:13:20.576777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.600093 (+ 23316us) service_pool.cc:227] Handling call
0812 19:13:22.630307 (+2030214us) spinlock_profiling.cc:243] Waited 2 s on lock 0x7b54002f2bd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:22.630541 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4597689088}
W20260812 19:13:22.688618 23452 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.626s user 0.006s sys 0.033s
I20260812 19:13:22.706310 23594 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:22.698503 23017 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 1.485s user 0.002s sys 0.012s
W20260812 19:13:22.751649 21287 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.645s user 0.019s sys 0.035s
I20260812 19:13:22.751816 23176 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:22.763906 23590 raft_consensus.cc:729] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:22.764230 23176 raft_consensus.cc:2726] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:22.635776 23575 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:22.765934 21287 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.831s user 0.020s sys 0.036s
W20260812 19:13:22.756227 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 1.92654e+06us
I20260812 19:13:22.756587 21320 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:22.863858 21327 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:22.698333 23422 raft_consensus.cc:686] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 3 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
W20260812 19:13:22.753490 23515 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.961s user 0.025s sys 0.101s
I20260812 19:13:22.839208 21240 pending_rounds.cc:76] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa: Aborting all ops after (but not including) 1
I20260812 19:13:22.891855 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1444) took 2411 ms. Trace:
W20260812 19:13:22.840349 23659 log.cc:924] Time spent T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.291s user 0.000s sys 0.002s
W20260812 19:13:22.871413 22045 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.696s user 0.008s sys 0.008s
I20260812 19:13:23.227341 23428 raft_consensus.cc:2726] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:22.883226 23605 consensus_queue.cc:237] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:22.890806 22011 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:23.226439 23658 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:23.232091 23674 raft_consensus.cc:2781] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:23.232631 23674 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:23.233062 23674 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:22.870523 23370 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:23.234170 21233 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1467) took 2481 ms (client timeout 2620 ms). Trace:
I20260812 19:13:23.000854 23455 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:23.234740 21233 rpcz_store.cc:269] 0812 19:13:20.752719 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:21.022102 (+269383us) service_pool.cc:227] Handling call
0812 19:13:23.233911 (+2211809us) spinlock_profiling.cc:243] Waited 31 ms on lock 0x7b540014ffd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:23.234099 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":71337216}
W20260812 19:13:22.875769 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 2.48598e+06us
W20260812 19:13:23.099114 21237 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57682 (request call id 183) took 2782 ms (client timeout 2587 ms). Trace:
I20260812 19:13:23.237253 21233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1466) took 2485 ms. Trace:
W20260812 19:13:23.237133 21237 rpcz_store.cc:269] 0812 19:13:20.220294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.220457 (+ 163us) service_pool.cc:227] Handling call
0812 19:13:23.001936 (+2781479us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54002c47d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:23.002182 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":288790272}
I20260812 19:13:23.237771 21233 rpcz_store.cc:276] 0812 19:13:20.752372 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:23.235931 (+2483559us) service_pool.cc:227] Handling call
0812 19:13:23.236465 (+ 534us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:23.236867 (+ 402us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:23.236895 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:23.236953 (+ 58us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:23.237000 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:23.237057 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:23.237171 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:23.232118 23658 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:23.238858 21237 raft_consensus.cc:1226] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:23.239653 21237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 187) took 2429 ms. Trace:
I20260812 19:13:22.868970 23536 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:22.893430 21243 raft_consensus.cc:1261] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260812 19:13:23.003422 21235 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1435) took 6567 ms (client timeout 2137 ms). Trace:
W20260812 19:13:22.871183 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 237067us
I20260812 19:13:23.240108 23623 raft_consensus.cc:3032] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Stepping down as leader of term 4
W20260812 19:13:22.840395 23668 log.cc:924] Time spent T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
W20260812 19:13:22.766314 23574 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.931s user 0.026s sys 0.112s
W20260812 19:13:23.206017 21286 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.009s sys 0.023s
W20260812 19:13:22.883592 21320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1444) took 2079 ms (client timeout 2413 ms). Trace:
I20260812 19:13:23.248695 21237 rpcz_store.cc:276] 0812 19:13:20.809890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:23.238135 (+2428245us) service_pool.cc:227] Handling call
0812 19:13:23.238551 (+ 416us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:23.239387 (+ 836us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:23.239591 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:23.274047 21286 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.833s user 0.009s sys 0.024s
W20260812 19:13:23.115505 22047 consensus_meta.cc:287] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 2.319s user 0.013s sys 0.009s
I20260812 19:13:22.873509 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1427) took 4867 ms. Trace:
I20260812 19:13:22.897044 22053 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 367) took 2160 ms. Trace:
W20260812 19:13:22.786909 23576 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.569s user 0.016s sys 0.022s
I20260812 19:13:23.045192 21240 pending_rounds.cc:97] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa: Aborting uncommitted NO_OP operation due to leader change: 2.2
W20260812 19:13:23.280239 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 389599us
W20260812 19:13:22.766474 23017 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 1.732s user 0.003s sys 0.012s
I20260812 19:13:22.838853 23642 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:23.268092 21241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1447) took 2780 ms. Trace:
W20260812 19:13:23.243502 21235 rpcz_store.cc:269] 0812 19:13:16.435951 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:16.651104 (+215153us) service_pool.cc:227] Handling call
0812 19:13:23.002908 (+6351804us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54002567d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:23.003162 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3177243904}
I20260812 19:13:23.197093 23257 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:22.867724 22056 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 370) took 2183 ms. Trace:
I20260812 19:13:22.875370 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:23.042868 22008 leader_election.cc:334] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.969s (SENT)
I20260812 19:13:23.272130 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1462) took 2670 ms. Trace:
W20260812 19:13:23.230190 21327 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1450) took 2343 ms (client timeout 2620 ms). Trace:
W20260812 19:13:22.997841 23646 log.cc:924] Time spent T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.126s user 0.001s sys 0.001s
I20260812 19:13:23.112298 23669 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:23.279721 22047 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:23.280205 23536 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:23.003644 23583 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:23.269332 22045 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.215s user 0.010s sys 0.019s
I20260812 19:13:23.277681 21318 rpcz_store.cc:276] 0812 19:13:18.005778 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.765079 (+759301us) service_pool.cc:227] Handling call
0812 19:13:18.799350 (+ 34271us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:18.898884 (+ 99534us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:18.898915 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:19.036170 (+137255us) log.cc:842] Serialized 26 byte log entry
0812 19:13:22.872026 (+3835856us) spinlock_profiling.cc:243] Waited 3.04 ms on lock 0x7b18000a9de0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d973582c 00007f52d6b6c9dc 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b89e22 00007f52d97275bb 00007f52d972d063 00007f52d972d411
0812 19:13:22.872565 (+ 539us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:22.872659 (+ 94us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:22.872809 (+ 150us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:22.872887 (+ 78us) raft_consensus.cc:1671] finished
0812 19:13:22.872900 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:22.873246 (+ 346us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1686,"raft.queue_time_us":51702,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":12721,"spinlock_wait_cycles":6981376,"thread_start_us":679546,"threads_started":1,"wal-append.queue_time_us":689461,"wal-append.run_cpu_time_us":2386,"wal-append.run_wall_time_us":1056724}
W20260812 19:13:23.287039 23432 consensus_meta.cc:287] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.745s user 0.020s sys 0.000s
I20260812 19:13:22.874749 23612 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:22.888893 23575 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:22.962779 23565 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:23.087877 23601 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:23.288091 22055 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 365) took 2933 ms. Trace:
I20260812 19:13:23.288142 23615 raft_consensus.cc:2781] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:22.862860 23590 consensus_queue.cc:260] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:22.877308 21321 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
W20260812 19:13:23.274093 21320 rpcz_store.cc:269] 0812 19:13:20.804450 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.838599 (+ 34149us) service_pool.cc:227] Handling call
0812 19:13:22.883438 (+2044839us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:23.111449 23661 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:22.887638 21323 raft_consensus.cc:2454] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:22.994355 22050 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 350) took 6593 ms. Trace:
W20260812 19:13:23.304944 21204 leader_election.cc:334] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Timed out: RequestConsensusVote RPC to 127.19.70.132:39579 timed out after 1.784s (SENT)
I20260812 19:13:23.305612 23592 raft_consensus.cc:686] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:23.305979 23593 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:23.312093 23287 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:23.355331 23603 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:23.313292 23443 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:22.960239 23594 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:22.881963 23451 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:23.278546 23658 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:23.315203 23534 raft_consensus.cc:3032] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Stepping down as leader of term 3
W20260812 19:13:23.300686 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 64035.2us
I20260812 19:13:23.322527 23477 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:23.413412 21233 raft_consensus.cc:1226] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:23.325533 21152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 210) took 3529 ms. Trace:
W20260812 19:13:23.328580 23643 log.cc:924] Time spent T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.785s user 0.003s sys 0.000s
I20260812 19:13:23.331725 23422 consensus_queue.cc:237] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:23.045430 21239 rpcz_store.cc:276] 0812 19:13:20.480376 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.480531 (+ 155us) service_pool.cc:227] Handling call
0812 19:13:20.481009 (+ 478us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:20.481339 (+ 330us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:20.665388 (+184049us) log.cc:842] Serialized 14 byte log entry
0812 19:13:22.891082 (+2225694us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:22.891264 (+ 182us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:22.891410 (+ 146us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:22.891530 (+ 120us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:22.891756 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41753,"prepare.queue_time_us":33380,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":91936,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":209,"spinlock_wait_cycles":33423104,"thread_start_us":2295100,"threads_started":3,"wal-append.queue_time_us":1882285}
I20260812 19:13:23.301429 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1451) took 2404 ms. Trace:
I20260812 19:13:23.336995 21243 rpcz_store.cc:276] 0812 19:13:20.578150 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:22.740409 (+2162259us) service_pool.cc:227] Handling call
0812 19:13:22.754178 (+ 13769us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:23.242927 (+488749us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:23.247927 (+ 5000us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:23.272267 23623 raft_consensus.cc:729] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:23.470873 23313 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:23.471663 23623 consensus_queue.cc:260] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:23.472380 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:23.472918 23612 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:23.299957 21240 raft_consensus.cc:2866] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:23.473207 21157 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:13:23.474753 23352 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:23.475621 23352 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:23.476382 23206 tablet_bootstrap.cc:492] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap starting.
I20260812 19:13:23.475883 23592 consensus_queue.cc:237] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:23.337373 21327 rpcz_store.cc:269] 0812 19:13:20.886445 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.891668 (+ 5223us) service_pool.cc:227] Handling call
0812 19:13:23.230051 (+2338383us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:23.346576 22008 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:22.875609 23551 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:23.300902 21241 rpcz_store.cc:276] 0812 19:13:20.487402 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.487584 (+ 182us) service_pool.cc:227] Handling call
0812 19:13:20.494161 (+ 6577us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:20.494432 (+ 271us) raft_consensus.cc:1520] Early marking committed up to index 2
0812 19:13:20.741149 (+246717us) log.cc:842] Serialized 14 byte log entry
0812 19:13:22.839259 (+2098110us) spinlock_profiling.cc:243] Waited 76.4 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:23.098268 (+259009us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:23.163002 (+ 64734us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:23.188672 (+ 25670us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:23.188903 (+ 231us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:23.267995 (+ 79092us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65362,"prepare.queue_time_us":20588,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":90733,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":136,"spinlock_wait_cycles":175643520,"thread_start_us":2399791,"threads_started":3,"wal-append.queue_time_us":1807176}
I20260812 19:13:22.970703 23436 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:23.361524 22051 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:13:23.334357 23452 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:23.277814 22053 rpcz_store.cc:276] 0812 19:13:20.616942 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.617172 (+ 230us) service_pool.cc:227] Handling call
0812 19:13:20.666149 (+ 48977us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:20.666405 (+ 256us) raft_consensus.cc:1520] Early marking committed up to index 6
0812 19:13:20.707178 (+ 40773us) log.cc:842] Serialized 14 byte log entry
0812 19:13:22.607326 (+1900148us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:22.669294 (+ 61968us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:22.738460 (+ 69166us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:22.754489 (+ 16029us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:22.777624 (+ 23135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35528,"prepare.queue_time_us":63937,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":319084,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":135,"spinlock_wait_cycles":208845056,"thread_start_us":1876217,"threads_started":2,"wal-append.queue_time_us":1835654}
I20260812 19:13:22.827433 23633 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:23.336396 22056 rpcz_store.cc:276] 0812 19:13:20.684618 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.684828 (+ 210us) service_pool.cc:227] Handling call
0812 19:13:20.685278 (+ 450us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:20.685571 (+ 293us) raft_consensus.cc:1520] Early marking committed up to index 6
0812 19:13:20.728380 (+ 42809us) log.cc:842] Serialized 14 byte log entry
0812 19:13:22.867039 (+2138659us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:22.867161 (+ 122us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:22.867270 (+ 109us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:22.867388 (+ 118us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:22.867613 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":199641,"prepare.queue_time_us":47040,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":190644,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":128,"thread_start_us":1868799,"threads_started":1,"wal-append.queue_time_us":2111990}
I20260812 19:13:23.473093 23623 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:23.472512 23585 raft_consensus.cc:3032] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:23.467774 21233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 188) took 2592 ms. Trace:
I20260812 19:13:23.565119 21233 rpcz_store.cc:276] 0812 19:13:20.822731 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:23.270045 (+2447314us) service_pool.cc:227] Handling call
0812 19:13:23.361631 (+ 91586us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:23.414256 (+ 52625us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:23.414492 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:23.116076 23596 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:23.473906 23477 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:23.468659 21328 rpcz_store.cc:276] 0812 19:13:20.887034 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:22.616523 (+1729489us) service_pool.cc:227] Handling call
0812 19:13:22.617290 (+ 767us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:22.618092 (+ 802us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:22.618133 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:22.690627 (+ 72494us) log.cc:842] Serialized 26 byte log entry
0812 19:13:22.954368 (+263741us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:22.954409 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:22.954497 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:23.290929 (+336432us) raft_consensus.cc:1671] finished
0812 19:13:23.290965 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:23.291171 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":91614,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":280,"thread_start_us":213773,"threads_started":1,"wal-append.queue_time_us":542619}
I20260812 19:13:23.414427 21152 rpcz_store.cc:276] 0812 19:13:19.795857 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:19.796053 (+ 196us) service_pool.cc:227] Handling call
0812 19:13:19.796424 (+ 371us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:20.771117 (+974693us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:23.324903 (+2553786us) spinlock_profiling.cc:243] Waited 970 ms on lock 0x7b54002931d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:23.325278 (+ 375us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2230153088}
I20260812 19:13:23.413404 23534 raft_consensus.cc:729] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:23.597030 23443 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:23.598649 23534 consensus_queue.cc:260] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:23.600309 23443 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:23.601244 23534 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:23.531284 23459 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:23.532033 21237 raft_consensus.cc:1261] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260812 19:13:23.532616 23352 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:23.473150 23658 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:23.626322 21237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 189) took 2803 ms. Trace:
I20260812 19:13:23.626921 21237 rpcz_store.cc:276] 0812 19:13:20.823062 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:23.338322 (+2515260us) service_pool.cc:227] Handling call
0812 19:13:23.395942 (+ 57620us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:23.619883 (+223941us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:23.626236 (+ 6353us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:23.533653 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:23.395891 22050 rpcz_store.cc:276] 0812 19:13:16.400795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:16.407732 (+ 6937us) service_pool.cc:227] Handling call
0812 19:13:18.300956 (+1893224us) spinlock_profiling.cc:243] Waited 27.3 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52bce 000000000034aea9 00007f52de972a30 00007f52dea387b4 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:18.301249 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:20.789081 (+2487832us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:22.993648 (+2204567us) spinlock_profiling.cc:243] Waited 2.38 s on lock 0x7b54004538d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:22.994090 (+ 442us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5528956160}
I20260812 19:13:23.537015 21112 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:23.537434 23579 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:23.538067 23502 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:23.537755 23287 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:23.538828 21236 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1457) took 2962 ms. Trace:
I20260812 19:13:23.539279 23473 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:23.538826 23565 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:23.537006 23515 consensus_queue.cc:1035] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260812 19:13:23.540206 23593 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:23.416132 23661 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:23.537101 21200 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:23.552109 23536 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:23.552852 23603 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:23.375263 22055 rpcz_store.cc:276] 0812 19:13:20.354161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.354831 (+ 670us) service_pool.cc:227] Handling call
0812 19:13:20.355132 (+ 301us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:20.355306 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:13:20.576672 (+221366us) log.cc:842] Serialized 14 byte log entry
0812 19:13:22.996773 (+2420101us) spinlock_profiling.cc:243] Waited 929 us on lock 0x7b180023bea0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b6c345 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b87155 00007f52d9a9e3d0 00007f52d9ab0576 00007f52d9ac10d7 00007f52d9ac1058
0812 19:13:23.189597 (+192824us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:23.216795 (+ 27198us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:23.287627 (+ 70832us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:23.287784 (+ 157us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:23.288004 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":99353,"prepare.queue_time_us":46442,"prepare.run_cpu_time_us":118,"prepare.run_wall_time_us":2336,"raft.queue_time_us":254582,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":140,"spinlock_wait_cycles":86533760,"thread_start_us":2340417,"threads_started":3,"wal-append.queue_time_us":2066316}
I20260812 19:13:23.559139 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:23.371943 23432 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:23.585245 21323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1449) took 2684 ms (client timeout 1814 ms). Trace:
I20260812 19:13:23.596833 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:23.492640 22047 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40624 (request call id 366) took 2800 ms (client timeout 2413 ms). Trace:
I20260812 19:13:23.531073 21238 raft_consensus.cc:1226] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
W20260812 19:13:23.603484 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 321625us
I20260812 19:13:23.594442 21203 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
W20260812 19:13:23.618486 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 542471us
I20260812 19:13:23.532519 23601 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:23.656318 21236 rpcz_store.cc:276] 0812 19:13:20.576454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:22.710030 (+2133576us) service_pool.cc:227] Handling call
0812 19:13:22.747282 (+ 37252us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:22.840847 (+ 93565us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:22.840885 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:22.841496 (+ 611us) log.cc:842] Serialized 26 byte log entry
0812 19:13:23.095572 (+254076us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:23.095624 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:23.095727 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:23.538450 (+442723us) raft_consensus.cc:1671] finished
0812 19:13:23.538482 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:23.538714 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":253491,"threads_started":1,"wal-append.queue_time_us":532648}
I20260812 19:13:23.564947 23585 raft_consensus.cc:729] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:23.471699 23313 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:23.689937 21323 rpcz_store.cc:269] 0812 19:13:20.835873 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.848552 (+ 12679us) service_pool.cc:227] Handling call
0812 19:13:23.500181 (+2651629us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:23.654955 23576 raft_consensus.cc:2726] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:22.544785 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:13:23.714278 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:23.522593 23669 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:23.697011 23473 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:23.715848 21288 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:13:23.730491 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:23.729425 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:23.618299 23452 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:23.747641 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 705007us
I20260812 19:13:23.727296 23612 raft_consensus.cc:2781] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:23.668419 23459 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:23.752959 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:23.759267 22046 raft_consensus.cc:1226] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:23.759671 22055 raft_consensus.cc:2396] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260812 19:13:23.760244 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:23.760528 21151 raft_consensus.cc:1226] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260812 19:13:23.760857 22046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40614 (request call id 398) took 3304 ms. Trace:
I20260812 19:13:23.761300 22046 rpcz_store.cc:276] 0812 19:13:20.456468 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:22.877187 (+2420719us) service_pool.cc:227] Handling call
0812 19:13:22.877818 (+ 631us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:23.759838 (+882020us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:23.760544 (+ 706us) spinlock_profiling.cc:243] Waited 831 ms on lock 0x7b540064afd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:23.760768 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1910916352}
I20260812 19:13:23.762590 23696 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:23.762750 23578 raft_consensus.cc:2781] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:23.763758 21156 raft_consensus.cc:1226] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:23.763931 23578 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:23.763542 23696 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:23.764393 23578 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:23.764552 23502 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:23.767027 21152 raft_consensus.cc:1226] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 1. Current term is 3. Ops: []
I20260812 19:13:23.767181 23502 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:23.767789 21153 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1705) took 3408 ms (client timeout 1531 ms). Trace:
I20260812 19:13:23.768082 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:23.768282 21153 rpcz_store.cc:269] 0812 19:13:20.359677 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.359929 (+ 252us) service_pool.cc:227] Handling call
0812 19:13:23.767333 (+3407404us) spinlock_profiling.cc:243] Waited 40.9 ms on lock 0x7b5400324954. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:23.767569 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":94164352}
I20260812 19:13:23.769074 21157 raft_consensus.cc:2379] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:23.761308 22052 raft_consensus.cc:2396] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260812 19:13:23.770543 22052 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40598 (request call id 585) took 3658 ms (client timeout 1969 ms). Trace:
I20260812 19:13:23.756995 23575 raft_consensus.cc:2726] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260812 19:13:23.772492 23688 log.cc:924] Time spent T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.088s user 0.000s sys 0.001s
I20260812 19:13:23.773077 23619 raft_consensus.cc:3032] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:23.773559 23619 raft_consensus.cc:729] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:23.774266 21158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1713) took 3329 ms. Trace:
I20260812 19:13:23.774684 23477 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:23.774951 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 237960us
I20260812 19:13:23.776114 23473 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:23.776196 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 421822us
I20260812 19:13:23.702338 23579 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:23.777226 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:23.770972 22052 rpcz_store.cc:269] 0812 19:13:20.111994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.112115 (+ 121us) service_pool.cc:227] Handling call
0812 19:13:23.759858 (+3647743us) spinlock_profiling.cc:243] Waited 1.16 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52b7c 00007f52d6b5273b 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:13:23.770462 (+ 10604us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4009472}
W20260812 19:13:23.778477 21321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1445) took 2973 ms (client timeout 1892 ms). Trace:
I20260812 19:13:23.770794 23287 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:23.795504 21321 rpcz_store.cc:269] 0812 19:13:20.804806 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.815939 (+ 11133us) service_pool.cc:227] Handling call
0812 19:13:23.778191 (+2962252us) spinlock_profiling.cc:243] Waited 1.82 s on lock 0x7b540016cbd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:23.778383 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4187995008}
I20260812 19:13:23.796149 23451 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:23.670368 23633 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:23.778479 23619 consensus_queue.cc:260] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
W20260812 19:13:23.779150 22011 connection.cc:232] RPC call timeout handler was delayed by -2.1923621819587424s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25870000000000004s for a response
W20260812 19:13:23.693612 22047 rpcz_store.cc:269] 0812 19:13:20.609094 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.625347 (+ 16253us) service_pool.cc:227] Handling call
0812 19:13:23.387677 (+2762330us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:23.803081 21244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1465) took 3046 ms. Trace:
W20260812 19:13:23.766284 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 229486us
W20260812 19:13:23.778530 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 643313us
I20260812 19:13:23.697847 23436 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:23.715739 21287 connection.cc:232] RPC call timeout handler was delayed by -6.0731014079647139s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
I20260812 19:13:23.758379 23612 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:23.752175 23680 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:23.752187 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:23.703715 23596 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:23.736253 23658 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:23.714761 23551 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:23.725628 23674 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:23.811946 21155 raft_consensus.cc:2396] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260812 19:13:23.807314 21244 rpcz_store.cc:276] 0812 19:13:20.752009 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:23.111776 (+2359767us) service_pool.cc:227] Handling call
0812 19:13:23.112332 (+ 556us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:23.112912 (+ 580us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:23.112950 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:23.113569 (+ 619us) log.cc:842] Serialized 26 byte log entry
0812 19:13:23.368203 (+254634us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:23.368247 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:23.368335 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:23.798525 (+430190us) raft_consensus.cc:1671] finished
0812 19:13:23.798554 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:23.798724 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1571,"raft.queue_time_us":5588,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":178,"thread_start_us":200804,"threads_started":1,"wal-append.queue_time_us":675609}
I20260812 19:13:23.894862 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:24.041026 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 86722.9us
I20260812 19:13:23.945039 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:24.147785 21152 raft_consensus.cc:1226] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
W20260812 19:13:23.901481 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 61537us
I20260812 19:13:23.901937 22053 raft_consensus.cc:1226] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:23.788290 21158 rpcz_store.cc:276] 0812 19:13:20.445020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.445399 (+ 379us) service_pool.cc:227] Handling call
0812 19:13:20.467024 (+ 21625us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:20.471957 (+ 4933us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:20.471998 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:22.988896 (+2516898us) spinlock_profiling.cc:243] Waited 3.46 ms on lock 0x7b2c0006b848. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d8495e11 00007f52d84a5442 00007f52d9ae1c11 00007f52d9aa658a 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772
0812 19:13:22.989833 (+ 937us) log.cc:842] Serialized 26 byte log entry
0812 19:13:23.210854 (+221021us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:23.210923 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:23.268754 (+ 57831us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:23.773799 (+505045us) raft_consensus.cc:1671] finished
0812 19:13:23.773820 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:23.774073 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":38593,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"spinlock_wait_cycles":7965056,"thread_start_us":156883,"threads_started":1,"wal-append.queue_time_us":496736}
W20260812 19:13:23.903185 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 58054.7us
I20260812 19:13:23.856030 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:23.794076 23017 log.cc:924] Time spent T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20260812 19:13:24.105548 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 117262us
I20260812 19:13:24.144255 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1446) took 3660 ms. Trace:
W20260812 19:13:23.814872 22008 leader_election.cc:334] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.799s (SENT)
W20260812 19:13:23.869591 23527 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.067s user 0.011s sys 0.022s
I20260812 19:13:24.146692 23473 consensus_queue.cc:1035] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
W20260812 19:13:24.147815 23647 log.cc:924] Time spent T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 1.598s user 0.000s sys 0.014s
W20260812 19:13:24.143782 23706 log.cc:924] Time spent T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.219s user 0.001s sys 0.000s
I20260812 19:13:24.191658 22052 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:24.200310 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:24.195191 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:24.160725 23612 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:24.199846 23674 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:24.197640 23451 consensus_queue.cc:1046] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260812 19:13:24.186874 21112 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:24.191210 21238 rpcz_store.cc:276] 0812 19:13:20.483615 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.483768 (+ 153us) service_pool.cc:227] Handling call
0812 19:13:20.486746 (+ 2978us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:23.712975 (+3226229us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:24.143826 (+430851us) spinlock_profiling.cc:243] Waited 2.98 s on lock 0x7b5400032354. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:24.144158 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6855333632}
I20260812 19:13:24.187466 23579 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:24.186862 23680 raft_consensus.cc:504] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:24.198014 22050 raft_consensus.cc:2379] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:23.924134 23697 raft_consensus.cc:2781] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260812 19:13:24.201709 21287 connection.cc:232] RPC call timeout handler was delayed by -6.0398014079546556s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1563s for a response
I20260812 19:13:23.911497 23594 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:24.171044 23432 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:24.209039 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 414371us
W20260812 19:13:24.247517 21287 connection.cc:232] RPC call timeout handler was delayed by -6.0020014080218971s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1605s for a response
W20260812 19:13:24.206272 21115 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.70.132:39579 timed out after 2.413s (SENT) [suppressed 6 similar messages]
I20260812 19:13:24.208487 22051 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 4.
I20260812 19:13:24.198330 21153 raft_consensus.cc:1226] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:24.122777 23603 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:24.208703 21200 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:24.202107 23663 consensus_queue.cc:1046] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260812 19:13:24.228576 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:24.250147 21157 raft_consensus.cc:3037] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:24.229110 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:24.229923 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:24.208351 23669 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:24.232205 23551 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:24.232951 21318 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:24.221606 23436 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:24.195989 23527 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.625s user 0.021s sys 0.028s
I20260812 19:13:24.201615 22053 raft_consensus.cc:1226] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:24.203591 23658 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:24.262261 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 56049.2us
I20260812 19:13:24.262151 21288 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
W20260812 19:13:24.262401 21287 connection.cc:232] RPC call timeout handler was delayed by -5.9624014080036432s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16490000000000002s for a response
I20260812 19:13:24.213558 23615 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:24.207687 23593 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:24.271828 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1412) took 7851 ms. Trace:
I20260812 19:13:24.208072 23601 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:24.273865 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 105382us
W20260812 19:13:24.275166 21287 connection.cc:232] RPC call timeout handler was delayed by -5.8526014080271125s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1771s for a response
W20260812 19:13:24.275610 21287 connection.cc:232] RPC call timeout handler was delayed by -5.82560140802525s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1801s for a response
I20260812 19:13:24.279618 23428 raft_consensus.cc:2781] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:24.280450 23428 raft_consensus.cc:686] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:24.281702 23428 consensus_queue.cc:237] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:24.245572 23697 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:24.284376 21287 connection.cc:232] RPC call timeout handler was delayed by -5.643801408004947s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20030000000000003s for a response
W20260812 19:13:24.290998 21287 connection.cc:232] RPC call timeout handler was delayed by -5.6312014079885557s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2017s for a response
I20260812 19:13:23.907836 23585 consensus_queue.cc:260] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:24.245951 23451 raft_consensus.cc:3032] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:24.224686 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:24.273283 23615 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:24.274024 23527 raft_consensus.cc:3032] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Stepping down as leader of term 2
W20260812 19:13:24.319674 21287 connection.cc:232] RPC call timeout handler was delayed by -5.62220140802674s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20270000000000002s for a response
I20260812 19:13:24.267220 22045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 371) took 3479 ms. Trace:
I20260812 19:13:24.295145 22051 raft_consensus.cc:1261] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260812 19:13:24.330519 23596 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:24.327260 22045 rpcz_store.cc:276] 0812 19:13:20.785433 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.788312 (+ 2879us) service_pool.cc:227] Handling call
0812 19:13:20.932456 (+144144us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:23.503126 (+2570670us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:23.514869 (+ 11743us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:23.554798 (+ 39929us) log.cc:842] Serialized 26 byte log entry
0812 19:13:23.700657 (+145859us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:23.703370 (+ 2713us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:23.729333 (+ 25963us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:24.245451 (+516118us) raft_consensus.cc:1671] finished
0812 19:13:24.245475 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:24.264905 (+ 19430us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1818866,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":324,"thread_start_us":2451083,"threads_started":2,"wal-append.queue_time_us":198242}
W20260812 19:13:24.332166 23701 log.cc:924] Time spent T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260812 19:13:24.339309 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:24.315522 23452 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:24.288399 23579 raft_consensus.cc:2726] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260812 19:13:24.345199 21287 connection.cc:232] RPC call timeout handler was delayed by -5.478201408055611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2187s for a response
I20260812 19:13:24.342842 23017 tablet_bootstrap.cc:492] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 19:13:24.342741 21200 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:24.289184 23593 raft_consensus.cc:2781] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:24.284679 23696 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:24.344323 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1464) took 3592 ms. Trace:
I20260812 19:13:24.273641 21322 rpcz_store.cc:276] 0812 19:13:16.345429 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:18.645156 (+2299727us) service_pool.cc:227] Handling call
0812 19:13:18.692933 (+ 47777us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:18.738238 (+ 45305us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:18.738271 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:18.769782 (+ 31511us) log.cc:842] Serialized 26 byte log entry
0812 19:13:19.212263 (+442481us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:19.212307 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:19.212384 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:24.013916 (+4801532us) spinlock_profiling.cc:243] Waited 8.1 ms on lock 0x7b24002012d8. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d69e2cf4 00007f52d69e0dfb 00007f52d6a6f559 00007f52d6a6f1e5 00007f52d6a6e740 00007f52d9755578 00007f52d9755cd8 00007f52d972d6e8 00007f52d9728aea 00007f52d9727f36
0812 19:13:24.151947 (+138031us) raft_consensus.cc:1671] finished
0812 19:13:24.151971 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:24.196184 (+ 44213us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32888,"raft.queue_time_us":35025,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":8481,"spinlock_wait_cycles":18624256,"thread_start_us":355908,"threads_started":1,"wal-append.queue_time_us":1072023}
W20260812 19:13:24.354228 21287 connection.cc:232] RPC call timeout handler was delayed by -4.785201407968998s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2957s for a response
W20260812 19:13:24.354660 21287 connection.cc:232] RPC call timeout handler was delayed by -4.2425014079781249s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.35600000000000004s for a response
I20260812 19:13:24.349769 21240 rpcz_store.cc:276] 0812 19:13:20.751654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:22.838077 (+2086423us) service_pool.cc:227] Handling call
0812 19:13:22.838730 (+ 653us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:23.474006 (+635276us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:23.474056 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:23.474588 (+ 532us) log.cc:842] Serialized 26 byte log entry
0812 19:13:23.746789 (+272201us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:23.746885 (+ 96us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:23.746964 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:24.343996 (+597032us) raft_consensus.cc:1671] finished
0812 19:13:24.344026 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:24.344231 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":271725,"threads_started":1,"wal-append.queue_time_us":757529}
W20260812 19:13:24.325356 23692 log.cc:924] Time spent T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.092s user 0.000s sys 0.001s
I20260812 19:13:24.295826 23633 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:24.360743 22008 leader_election.cc:334] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.891s (SENT)
I20260812 19:13:24.321530 23451 raft_consensus.cc:729] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
W20260812 19:13:24.368752 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 187266us
W20260812 19:13:24.360463 21204 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.056s sys 0.084s
W20260812 19:13:24.369959 21204 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.515s user 0.056s sys 0.085s
W20260812 19:13:24.370246 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 621205us
W20260812 19:13:24.371084 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 78966.6us
I20260812 19:13:24.327215 23527 raft_consensus.cc:729] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:24.375197 21203 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:24.353025 23593 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:24.317451 23658 consensus_queue.cc:1046] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260812 19:13:24.292451 23697 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:24.348678 23459 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:24.369788 23585 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:24.395538 21157 raft_consensus.cc:2454] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:24.398458 23680 leader_election.cc:288] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:24.399674 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:24.398495 22051 raft_consensus.cc:1226] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:24.372946 23681 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:24.414301 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:24.417088 21204 leader_election.cc:334] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Timed out: RequestConsensusVote RPC to 127.19.70.132:39579 timed out after 1.861s (SENT)
I20260812 19:13:24.390969 23371 raft_consensus.cc:2726] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:24.430702 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:24.430846 23451 consensus_queue.cc:260] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:24.431905 21327 raft_consensus.cc:2379] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 4: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:24.432370 23451 raft_consensus.cc:3037] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:24.433075 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:24.434029 21327 raft_consensus.cc:2379] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 3: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:24.434057 23681 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:24.438228 21237 raft_consensus.cc:1261] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260812 19:13:24.438834 21241 pending_rounds.cc:76] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa: Aborting all ops after (but not including) 2
I20260812 19:13:24.439213 21241 pending_rounds.cc:97] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260812 19:13:24.439497 21241 raft_consensus.cc:2866] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:24.439653 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:24.436779 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:24.441720 21327 raft_consensus.cc:2396] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260812 19:13:24.442806 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:24.400645 23633 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:24.422904 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:24.422776 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:24.448478 21233 raft_consensus.cc:1226] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260812 19:13:24.448685 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:24.449096 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:24.449347 21244 raft_consensus.cc:1226] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260812 19:13:24.410014 21318 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
W20260812 19:13:24.451841 22008 connection.cc:232] RPC call timeout handler was delayed by -2.4985988599946722s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1847s for a response
W20260812 19:13:24.452466 22008 connection.cc:232] RPC call timeout handler was delayed by -1.8014918559929356s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18700000000000003s for a response
W20260812 19:13:24.452845 22008 connection.cc:232] RPC call timeout handler was delayed by -1.10805860592518s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25870000000000004s for a response
I20260812 19:13:24.449765 21327 raft_consensus.cc:2454] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:24.456019 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:24.381472 23593 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:24.459928 23623 consensus_meta.cc:287] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.929s user 0.005s sys 0.008s
I20260812 19:13:24.459823 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:24.461002 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:24.461350 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:24.422425 23651 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.808s user 0.038s sys 0.065s
W20260812 19:13:24.464243 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 104666us
I20260812 19:13:24.449777 22047 raft_consensus.cc:2396] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260812 19:13:24.465853 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:24.467453 21238 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:24.465713 23676 consensus_queue.cc:1046] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260812 19:13:24.469787 23608 consensus_queue.cc:1035] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.100s
I20260812 19:13:24.470739 23372 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:24.472043 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:24.473603 21324 raft_consensus.cc:2396] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260812 19:13:24.469018 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:24.466359 23651 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.852s user 0.039s sys 0.066s
I20260812 19:13:24.485612 22052 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:24.493021 21318 raft_consensus.cc:2354] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:13:24.489657 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:24.457721 23658 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:24.496337 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:24.497068 23681 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:24.497326 21319 raft_consensus.cc:2354] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:13:24.498557 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:24.498557 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:24.503041 21319 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:24.501670 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:24.478354 23633 raft_consensus.cc:686] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:24.512532 21233 raft_consensus.cc:1261] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260812 19:13:24.512974 23017 tablet_bootstrap.cc:492] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:13:24.513803 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:24.514115 23017 ts_tablet_manager.cc:1397] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 5.928s user 0.051s sys 0.067s
W20260812 19:13:24.518164 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 102186us
I20260812 19:13:24.521706 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:24.567703 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 59736us
I20260812 19:13:24.574470 21319 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:24.567013 21236 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:24.580698 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:24.581988 21318 raft_consensus.cc:2379] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 3: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:24.584643 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:24.578423 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:24.590911 22052 raft_consensus.cc:2379] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:24.589838 21319 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:24.595074 21112 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:24.597956 22008 leader_election.cc:334] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.870s (SENT)
I20260812 19:13:24.599246 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:24.585403 23578 consensus_meta.cc:287] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.820s user 0.014s sys 0.003s
I20260812 19:13:24.516695 23658 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:24.624334 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:24.625595 21319 raft_consensus.cc:2362] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3. Re-sending same reply.
W20260812 19:13:24.671258 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 73358.4us
I20260812 19:13:24.619457 23578 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:24.705688 21155 raft_consensus.cc:1226] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260812 19:13:24.635814 23459 raft_consensus.cc:2781] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:24.723457 23459 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:24.723994 23459 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:24.627609 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:24.727389 22047 raft_consensus.cc:2396] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260812 19:13:24.728214 23638 log.cc:924] Time spent T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 2.084s user 0.016s sys 0.031s
W20260812 19:13:24.708072 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 67760.6us
I20260812 19:13:24.613665 23676 consensus_queue.cc:1046] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20260812 19:13:24.731309 23676 raft_consensus.cc:3032] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:24.731750 23676 raft_consensus.cc:729] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:24.731904 23608 raft_consensus.cc:3032] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:24.732609 23608 raft_consensus.cc:729] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:24.733166 23619 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:24.732885 23676 consensus_queue.cc:260] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:24.737579 23452 raft_consensus.cc:3032] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:24.738077 23452 raft_consensus.cc:729] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:24.713163 23578 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:24.714706 22048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40614 (request call id 393) took 4598 ms. Trace:
I20260812 19:13:24.715896 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:24.719535 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:24.721998 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 148848us
I20260812 19:13:24.745285 22048 rpcz_store.cc:276] 0812 19:13:20.010034 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.013333 (+ 3299us) service_pool.cc:227] Handling call
0812 19:13:20.119776 (+106443us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:20.178953 (+ 59177us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:24.607435 (+4428482us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b5400611554. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:24.607900 (+ 465us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32160640}
I20260812 19:13:24.756978 23658 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:24.742177 23676 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:24.761279 21318 raft_consensus.cc:1226] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:24.747524 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:24.572957 23527 consensus_queue.cc:260] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:24.744424 22008 leader_election.cc:334] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.587s (SENT)
I20260812 19:13:24.780685 21319 raft_consensus.cc:2396] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260812 19:13:24.791899 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:24.797542 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:24.734027 23608 consensus_queue.cc:260] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:24.805662 22055 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.000s sys 0.030s
W20260812 19:13:24.808878 22055 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s user 0.000s sys 0.031s
I20260812 19:13:24.808115 21324 raft_consensus.cc:2396] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260812 19:13:24.807786 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:24.812098 22008 leader_election.cc:334] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.547s (SENT)
I20260812 19:13:24.812616 22008 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:24.816231 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:24.818858 21203 leader_election.cc:398] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3.
I20260812 19:13:24.819387 21203 leader_election.cc:401] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260812 19:13:24.819096 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:24.761183 22050 raft_consensus.cc:2362] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6. Re-sending same reply.
I20260812 19:13:24.823918 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:24.833941 22011 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.012s sys 0.024s
W20260812 19:13:24.834741 22011 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.509s user 0.013s sys 0.024s
W20260812 19:13:24.835057 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 607897us
I20260812 19:13:24.835443 21244 raft_consensus.cc:2396] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260812 19:13:24.832891 21236 raft_consensus.cc:2362] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5. Re-sending same reply.
I20260812 19:13:24.831913 21322 raft_consensus.cc:2454] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 2.
I20260812 19:13:24.842267 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:24.850083 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:24.851387 21322 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:24.840459 22055 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 590) took 1502 ms. Trace:
I20260812 19:13:24.852756 22055 rpcz_store.cc:276] 0812 19:13:23.322278 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:23.760441 (+438163us) service_pool.cc:227] Handling call
0812 19:13:23.780336 (+ 19895us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:23.780765 (+ 429us) raft_consensus.cc:1520] Early marking committed up to index 6
0812 19:13:24.034211 (+253446us) log.cc:842] Serialized 14 byte log entry
0812 19:13:24.824400 (+790189us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:24.824489 (+ 89us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:24.824556 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:24.824654 (+ 98us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:24.824808 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5505,"prepare.queue_time_us":555385,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":74,"raft.queue_time_us":27918,"thread_start_us":836524,"threads_started":2}
I20260812 19:13:24.822858 23527 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:24.843472 21112 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:24.819968 23608 raft_consensus.cc:3037] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:24.761914 23452 consensus_queue.cc:260] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:24.864360 21238 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:24.760762 23633 consensus_queue.cc:237] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:24.852815 23372 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:24.870800 23254 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.081s user 0.014s sys 0.048s
I20260812 19:13:24.870407 23651 consensus_queue.cc:1046] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 19:13:24.871048 23592 raft_consensus.cc:2726] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:24.871416 21321 raft_consensus.cc:2454] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
W20260812 19:13:24.872109 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 57226.3us
I20260812 19:13:24.867977 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:24.867837 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:24.843968 22046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 591) took 1508 ms. Trace:
I20260812 19:13:24.895232 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:24.896669 21236 raft_consensus.cc:2396] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260812 19:13:24.879207 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 67132us
I20260812 19:13:24.870296 23452 raft_consensus.cc:3037] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:24.886582 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:24.853680 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:24.870965 23578 raft_consensus.cc:1050] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:13:24.890156 21153 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
W20260812 19:13:24.881356 23254 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.131s user 0.014s sys 0.049s
I20260812 19:13:24.876789 23623 raft_consensus.cc:2726] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:24.870822 23371 raft_consensus.cc:2726] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3.
I20260812 19:13:24.908845 23425 consensus_queue.cc:1035] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
W20260812 19:13:24.924448 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 136091us
I20260812 19:13:24.911149 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:24.930358 21238 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:24.921718 23578 raft_consensus.cc:1054] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: Unable to promote non-voter 63db10063a7d4eb2a9cc9dc952654706: Illegal state: Leader has not yet committed an operation in its own term
I20260812 19:13:24.938299 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:24.889055 23432 consensus_queue.cc:1046] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260812 19:13:24.942381 23578 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:24.933637 23590 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:24.958043 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:24.916410 22008 leader_election.cc:334] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.576s (SENT)
I20260812 19:13:24.934943 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:24.947405 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:24.962730 22008 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:24.963958 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 64562.8us
I20260812 19:13:24.964871 23592 raft_consensus.cc:2726] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260812 19:13:24.966183 23615 consensus_meta.cc:287] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.608s user 0.009s sys 0.000s
I20260812 19:13:24.967020 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:24.893872 22046 rpcz_store.cc:276] 0812 19:13:23.335015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:23.762593 (+427578us) service_pool.cc:227] Handling call
0812 19:13:23.924970 (+162377us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:24.148120 (+223150us) raft_consensus.cc:1520] Early marking committed up to index 6
0812 19:13:24.378831 (+230711us) log.cc:842] Serialized 14 byte log entry
0812 19:13:24.797296 (+418465us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:24.802169 (+ 4873us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:24.808373 (+ 6204us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:24.814177 (+ 5804us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:24.830591 (+ 16414us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":188406,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":60,"raft.queue_time_us":11291,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":22770,"thread_start_us":613197,"threads_started":2}
I20260812 19:13:24.909992 23372 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:24.969039 21321 raft_consensus.cc:1226] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:24.915647 21115 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:24.957942 23578 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:24.979269 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:24.980197 21237 raft_consensus.cc:2354] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5.
I20260812 19:13:24.970476 23372 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:24.983382 23697 consensus_meta.cc:287] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.565s user 0.008s sys 0.004s
I20260812 19:13:24.984149 23697 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:24.968297 21236 raft_consensus.cc:1226] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5. Ops: []
W20260812 19:13:24.985918 23593 consensus_meta.cc:287] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.526s user 0.013s sys 0.000s
I20260812 19:13:24.986800 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:24.986450 23593 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:24.975263 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:24.988420 21235 raft_consensus.cc:2379] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:24.988574 21327 raft_consensus.cc:2354] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:13:24.989542 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:24.974766 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:24.991915 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:24.993314 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:24.972693 21324 raft_consensus.cc:1226] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:25.002663 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:24.998811 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 67348.3us
I20260812 19:13:25.003423 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:24.951090 21287 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:24.977674 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:25.016819 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:25.023401 21328 raft_consensus.cc:1226] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:25.027861 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:24.992327 23593 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:25.001860 23578 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:25.034991 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:25.002260 23661 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:24.953778 23017 raft_consensus.cc:348] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:25.010799 23313 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:25.044322 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:25.044796 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:25.045506 21156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 217) took 4228 ms. Trace:
I20260812 19:13:25.041541 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:25.049407 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 137419us
W20260812 19:13:25.040191 23585 consensus_meta.cc:287] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.627s user 0.009s sys 0.004s
I20260812 19:13:25.047840 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:25.051476 21243 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:13:25.055662 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:25.051723 21156 rpcz_store.cc:276] 0812 19:13:20.811676 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:20.811889 (+ 213us) service_pool.cc:227] Handling call
0812 19:13:20.814043 (+ 2154us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:23.764285 (+2950242us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:25.029783 (+1265498us) spinlock_profiling.cc:243] Waited 2.92 s on lock 0x7b54001a3454. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:25.040034 (+ 10251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6708674304}
W20260812 19:13:25.058347 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 193439us
I20260812 19:13:25.060931 21244 raft_consensus.cc:2396] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260812 19:13:25.055079 21115 consensus_peers.cc:487] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 -> Peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579): Couldn't send request to peer 63db10063a7d4eb2a9cc9dc952654706. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260812 19:13:25.059546 21235 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 4.
I20260812 19:13:25.067239 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:25.068250 21324 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 6: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:13:25.070994 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 50335.3us
I20260812 19:13:25.050757 23017 raft_consensus.cc:729] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:13:25.053246 23477 raft_consensus.cc:2726] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:25.022225 23615 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:25.083297 21238 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:25.087514 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:25.095181 21318 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:25.056347 23572 raft_consensus.cc:2781] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:25.101035 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:25.102190 21324 raft_consensus.cc:2396] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260812 19:13:25.102242 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:25.108701 21033 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58998 (request call id 140) took 11792 ms (client timeout 15000 ms). Trace:
W20260812 19:13:25.109262 21033 rpcz_store.cc:269] 0812 19:13:13.274399 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:13.274744 (+ 345us) service_pool.cc:227] Handling call
0812 19:13:25.058012 (+11783268us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0812 19:13:14.565006 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0812 19:13:14.871434 (+306428us) write_op.cc:432] Acquiring schema lock in shared mode
0812 19:13:14.887134 (+ 15700us) write_op.cc:435] Acquired schema lock
0812 19:13:14.899031 (+ 11897us) tablet.cc:660] Decoding operations
0812 19:13:15.329172 (+430141us) write_op.cc:620] Acquiring the partition lock for write op
0812 19:13:15.374006 (+ 44834us) write_op.cc:641] Partition lock acquired for write op
0812 19:13:15.379580 (+ 5574us) tablet.cc:683] Acquiring locks for 4 operations
0812 19:13:15.449836 (+ 70256us) tablet.cc:699] Row locks acquired
0812 19:13:15.449895 (+ 59us) write_op.cc:260] PREPARE: finished
0812 19:13:15.450126 (+ 231us) write_op.cc:270] Start()
0812 19:13:15.450320 (+ 194us) write_op.cc:276] Timestamp: P: 1786561995450038 usec, L: 0
0812 19:13:15.450347 (+ 27us) op_driver.cc:331] REPLICATION: starting
0812 19:13:15.695168 (+244821us) log.cc:842] Serialized 1686 byte log entry
0812 19:13:18.479798 (+2784630us) op_driver.cc:444] REPLICATION: finished
0812 19:13:19.202878 (+723080us) write_op.cc:301] APPLY: starting
0812 19:13:19.214114 (+ 11236us) tablet.cc:1365] starting BulkCheckPresence
0812 19:13:19.303450 (+ 89336us) tablet.cc:1368] finished BulkCheckPresence
0812 19:13:19.394029 (+ 90579us) tablet.cc:1370] starting ApplyRowOperation cycle
0812 19:13:19.749972 (+355943us) tablet.cc:1381] finished ApplyRowOperation cycle
0812 19:13:19.819335 (+ 69363us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0812 19:13:19.835102 (+ 15767us) write_op.cc:312] APPLY: finished
0812 19:13:19.883053 (+ 47951us) log.cc:842] Serialized 41 byte log entry
0812 19:13:20.090714 (+207661us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:13:20.090868 (+ 154us) write_op.cc:454] Released schema lock
0812 19:13:20.090920 (+ 52us) write_op.cc:341] FINISH: Updating metrics
0812 19:13:20.091269 (+ 349us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:13:20.091302 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":90163,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":402668,"apply.run_cpu_time_us":18119,"apply.run_wall_time_us":915946,"num_ops":4,"prepare.queue_time_us":473970,"prepare.run_cpu_time_us":54385,"prepare.run_wall_time_us":2153665,"raft.queue_time_us":1201310,"raft.run_cpu_time_us":89844,"raft.run_wall_time_us":386536,"replication_time_us":2950957,"thread_start_us":2647036,"threads_started":5,"wal-append.queue_time_us":1487303,"wal-append.run_cpu_time_us":84464,"wal-append.run_wall_time_us":3231223}]]}
I20260812 19:13:25.114754 23572 raft_consensus.cc:686] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:25.115795 23615 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:25.115994 23572 consensus_queue.cc:237] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:25.116638 22045 raft_consensus.cc:1226] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:25.100838 23371 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:25.119120 23371 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:25.121438 23371 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:25.122495 21156 raft_consensus.cc:1226] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:25.122646 23017 consensus_queue.cc:260] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:25.123808 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:25.124643 21241 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 6: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
W20260812 19:13:25.130704 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 179712us
I20260812 19:13:25.133319 23017 ts_tablet_manager.cc:1428] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 0.619s user 0.012s sys 0.004s
I20260812 19:13:25.133662 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:25.133796 23017 ts_tablet_manager.cc:1460] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Tablet startup took 619ms
I20260812 19:13:25.133883 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:25.134580 21324 raft_consensus.cc:2379] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:25.134706 21156 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:25.136193 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:25.136341 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 44903.1us
W20260812 19:13:25.098871 23612 consensus_meta.cc:287] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.877s user 0.012s sys 0.002s
I20260812 19:13:25.138767 21112 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:25.136826 21238 raft_consensus.cc:2362] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4. Re-sending same reply.
I20260812 19:13:25.133515 23621 raft_consensus.cc:3032] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:25.158744 22011 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:25.136404 23697 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:25.165238 21243 raft_consensus.cc:1261] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260812 19:13:25.166661 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:25.168036 21323 raft_consensus.cc:2454] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:25.168524 21322 raft_consensus.cc:2396] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260812 19:13:25.170194 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:25.177011 21241 raft_consensus.cc:2379] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:25.180640 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:25.181529 21238 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
W20260812 19:13:25.184405 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 82383.8us
I20260812 19:13:25.188061 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:25.192615 22011 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:25.160858 23621 raft_consensus.cc:729] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:25.142035 23612 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:25.178133 23254 raft_consensus.cc:2726] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:25.207705 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:25.209008 21324 raft_consensus.cc:2396] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260812 19:13:25.170701 23372 consensus_queue.cc:1046] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260812 19:13:25.195232 23615 raft_consensus.cc:2781] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:25.220502 21233 raft_consensus.cc:2379] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 4: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:25.228952 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:25.230368 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:25.231475 21238 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:25.235396 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:25.237576 23592 consensus_queue.cc:1046] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260812 19:13:25.243784 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:25.244421 23372 raft_consensus.cc:3032] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:25.244796 21153 raft_consensus.cc:2396] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260812 19:13:25.244942 23372 raft_consensus.cc:729] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:25.245985 23727 raft_consensus.cc:2781] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:25.246585 23727 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:25.247107 23727 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:25.249873 23565 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:25.250946 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 46667.9us
W20260812 19:13:25.134353 23017 ts_tablet_manager.cc:1462] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Trace:
0812 19:12:58.899097 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:12:58.899424 (+ 327us) service_pool.cc:227] Handling call
0812 19:12:59.300618 (+401194us) ts_tablet_manager.cc:928] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
0812 19:13:03.480765 (+4180147us) tablet_copy_client.cc:361] Tablet copy session begun
0812 19:13:04.042740 (+561975us) tablet_metadata.cc:641] Metadata flushed
0812 19:13:04.061866 (+ 19126us) tablet_copy_client.cc:490] Wrote new tablet metadata
0812 19:13:05.577834 (+1515968us) tablet_copy_client.cc:411] Wrote new consensus metadata
0812 19:13:05.636773 (+ 58939us) tablet_replica.cc:205] Creating consensus instance
0812 19:13:05.684730 (+ 47957us) ts_tablet_manager.cc:983] Replica registered.
0812 19:13:05.731453 (+ 46723us) inbound_call.cc:173] Queueing success response
0812 19:13:18.586248 (+12854795us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0812 19:13:24.514590 (+5928342us) ts_tablet_manager.cc:1429] Starting tablet replica
0812 19:13:24.869073 (+354483us) tablet_replica.cc:259] Starting instrumentation
0812 19:13:24.950910 (+ 81837us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":464,"mutex_wait_us":19088,"raft.queue_time_us":8823,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":16986,"tablet-copy.queue_time_us":236491,"tablet-download-pool.queue_time_us":37458,"tablet-download-pool.run_cpu_time_us":4068,"tablet-download-pool.run_wall_time_us":10783133,"thread_start_us":475250,"threads_started":2}
W20260812 19:13:25.253391 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 101857us
I20260812 19:13:25.254220 21287 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:25.255726 21287 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:25.257267 21158 raft_consensus.cc:1226] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260812 19:13:25.258224 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:25.258767 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:25.258955 21154 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:25.259888 23605 consensus_queue.cc:1035] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:13:25.202551 23459 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:25.262272 21151 raft_consensus.cc:1226] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:25.263684 23459 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:25.264582 21152 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 6: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:13:25.265202 23676 consensus_meta.cc:287] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.506s user 0.010s sys 0.004s
I20260812 19:13:25.267061 23615 raft_consensus.cc:686] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:25.252753 21112 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:25.268664 23615 consensus_queue.cc:237] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:25.311163 21112 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:25.295260 23452 raft_consensus.cc:2741] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260812 19:13:25.320739 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:25.302915 23658 raft_consensus.cc:2726] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:25.332564 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:25.320459 21320 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:25.321094 21236 raft_consensus.cc:2396] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260812 19:13:25.332012 21112 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:25.335034 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 79437.5us
I20260812 19:13:25.305369 23459 raft_consensus.cc:2781] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:25.296267 23565 consensus_queue.cc:1035] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.042s
I20260812 19:13:25.342828 21200 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:25.317508 23633 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:25.350412 22008 leader_election.cc:334] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.686s (SENT)
I20260812 19:13:25.351682 22008 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:25.329885 23676 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:25.334096 23652 raft_consensus.cc:2781] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:13:25.357151 21153 raft_consensus.cc:2379] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:25.246325 23372 consensus_queue.cc:260] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:25.359578 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:25.359763 21287 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:25.363277 21152 raft_consensus.cc:1261] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260812 19:13:25.366595 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:25.374816 21154 raft_consensus.cc:1261] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260812 19:13:25.376653 21321 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:25.347100 23459 raft_consensus.cc:686] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:25.375254 21154 pending_rounds.cc:76] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059: Aborting all ops after (but not including) 2
I20260812 19:13:25.387977 21154 pending_rounds.cc:97] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260812 19:13:25.388384 21154 raft_consensus.cc:2866] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:25.378978 23605 raft_consensus.cc:2781] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:25.378234 23658 raft_consensus.cc:2726] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 19:13:25.398820 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 44608.1us
I20260812 19:13:25.330308 23621 consensus_queue.cc:260] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:25.401289 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:25.361581 23633 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:25.406126 22056 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:13:25.427703 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:25.428658 22051 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:25.399595 23372 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:25.401854 23608 raft_consensus.cc:2781] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260812 19:13:25.440589 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 97861.8us
I20260812 19:13:25.400480 23590 consensus_queue.cc:1046] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260812 19:13:25.444710 23590 raft_consensus.cc:3032] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:25.445392 23590 raft_consensus.cc:729] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:25.446417 22055 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 LEARNER]: Advancing to term 3
I20260812 19:13:25.446478 23744 raft_consensus.cc:2726] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:25.426003 23612 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:25.447957 23737 log.cc:924] Time spent T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260812 19:13:25.444895 21321 raft_consensus.cc:1261] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260812 19:13:25.391359 23676 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:25.444764 21149 raft_consensus.cc:1261] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260812 19:13:25.462780 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:25.436949 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 58951.8us
I20260812 19:13:25.487708 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:25.425660 23605 raft_consensus.cc:686] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:13:25.490172 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 98482.2us
W20260812 19:13:25.489945 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 47748.5us
I20260812 19:13:25.491446 21329 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:25.496800 21200 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
W20260812 19:13:25.455286 23733 log.cc:924] Time spent T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260812 19:13:25.434227 23621 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:25.460713 23734 log.cc:924] Time spent T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260812 19:13:25.485643 23652 raft_consensus.cc:686] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:25.440698 23608 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:25.506625 21112 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:25.461606 22045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 385) took 1004 ms. Trace:
I20260812 19:13:25.431663 23592 consensus_queue.cc:1046] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20260812 19:13:25.535050 21287 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:25.503513 23633 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:25.514393 23722 raft_consensus.cc:2781] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260812 19:13:25.537163 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 192321us
I20260812 19:13:25.521857 23608 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:25.540598 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 310674us
I20260812 19:13:25.537372 21203 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:25.545334 21200 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
W20260812 19:13:25.552162 23527 consensus_meta.cc:287] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.686s user 0.008s sys 0.008s
I20260812 19:13:25.550275 22055 raft_consensus.cc:1261] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEARNER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260812 19:13:25.520525 22045 rpcz_store.cc:276] 0812 19:13:24.456857 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:24.471828 (+ 14971us) service_pool.cc:227] Handling call
0812 19:13:24.472160 (+ 332us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:25.117050 (+644890us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:25.461225 (+344175us) spinlock_profiling.cc:243] Waited 644 ms on lock 0x7b5400452254. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:25.461501 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1481590144}
I20260812 19:13:25.540961 23722 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:25.559433 22053 raft_consensus.cc:2454] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:25.559687 23722 raft_consensus.cc:3037] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:25.548725 23658 raft_consensus.cc:2726] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:25.460318 23590 consensus_queue.cc:260] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:25.457624 23459 consensus_queue.cc:237] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:25.576085 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:25.576916 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 85019.7us
I20260812 19:13:25.583936 23590 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:25.584792 21155 raft_consensus.cc:1226] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260812 19:13:25.580600 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:25.592898 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:25.600150 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 89950.6us
I20260812 19:13:25.572649 23612 raft_consensus.cc:2726] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:25.579547 23633 raft_consensus.cc:2781] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:25.606928 22055 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:25.610293 21155 raft_consensus.cc:1226] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260812 19:13:25.611970 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 51693.6us
I20260812 19:13:25.592203 23697 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:25.572782 23658 consensus_queue.cc:1035] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260812 19:13:25.573207 23750 consensus_queue.cc:1046] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260812 19:13:25.616143 23633 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:25.638377 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 66538.4us
I20260812 19:13:25.645314 22048 raft_consensus.cc:1226] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEADER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:25.622332 23754 raft_consensus.cc:2781] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:25.629761 23729 consensus_queue.cc:1046] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260812 19:13:25.682765 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:25.683174 23727 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:25.685633 23727 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:25.592465 23605 consensus_queue.cc:237] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:25.686805 23762 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:25.687553 23762 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:25.690097 23762 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:25.690846 23744 raft_consensus.cc:3032] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:25.651275 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:25.691344 23744 raft_consensus.cc:729] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
W20260812 19:13:25.702409 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 58433.1us
I20260812 19:13:25.637753 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:25.708801 21329 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
W20260812 19:13:25.692063 23741 log.cc:924] Time spent T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20260812 19:13:25.671980 23676 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:25.722718 23754 raft_consensus.cc:686] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:13:25.723532 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 48808.4us
W20260812 19:13:25.731398 23206 log.cc:924] Time spent T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
W20260812 19:13:25.733191 21034 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48448 (request call id 257) took 11699 ms (client timeout 15000 ms). Trace:
I20260812 19:13:25.627359 23697 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:25.651298 23633 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:25.750627 21034 rpcz_store.cc:269] 0812 19:13:14.033942 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:14.034272 (+ 330us) service_pool.cc:227] Handling call
0812 19:13:25.732950 (+11698678us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4428697,"reported_tablets":5}
I20260812 19:13:25.705917 23713 raft_consensus.cc:3032] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEADER]: Stepping down as leader of term 4
W20260812 19:13:25.769449 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 50560.3us
I20260812 19:13:25.771672 21329 raft_consensus.cc:1261] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260812 19:13:25.771413 21200 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:25.773353 21200 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:25.765939 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:25.775789 21151 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:25.618170 23652 consensus_queue.cc:237] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:25.795665 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:25.704787 23727 consensus_queue.cc:1046] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
W20260812 19:13:25.800691 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 204819us
W20260812 19:13:25.803089 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 298902us
I20260812 19:13:25.740260 23750 raft_consensus.cc:3032] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:25.794529 22046 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 6: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:25.764365 23713 raft_consensus.cc:729] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:25.915059 21203 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:25.931445 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:25.951726 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:25.932583 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:25.953150 21323 raft_consensus.cc:2396] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260812 19:13:25.933715 22053 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40624 (request call id 381) took 1418 ms. Trace:
I20260812 19:13:25.954931 22053 rpcz_store.cc:276] 0812 19:13:24.449977 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:24.450121 (+ 144us) service_pool.cc:227] Handling call
0812 19:13:25.867944 (+1417823us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540064c8d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:25.868081 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2465600896}
W20260812 19:13:25.955009 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 256446us
I20260812 19:13:25.947139 23602 raft_consensus.cc:3032] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:25.950103 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:25.960636 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 72872us
I20260812 19:13:25.961692 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:25.949026 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:25.962670 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 207049us
I20260812 19:13:25.963053 21328 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:25.964423 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:25.966357 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:25.967464 22047 raft_consensus.cc:2379] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 4: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:25.968400 23428 raft_consensus.cc:2781] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:25.970458 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:25.975167 22055 raft_consensus.cc:1261] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:25.973052 22045 raft_consensus.cc:1226] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:25.974751 23313 raft_consensus.cc:3032] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:25.977378 23313 raft_consensus.cc:729] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:25.977775 22055 raft_consensus.cc:1261] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:25.978574 23428 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:25.942468 23608 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:25.979121 23428 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:25.984181 21319 raft_consensus.cc:1261] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:25.997259 21115 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:26.003707 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:26.003880 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:26.004958 21323 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:26.005155 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:25.929682 23750 raft_consensus.cc:729] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:13:26.018081 21320 consensus_meta.cc:287] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.683s user 0.008s sys 0.004s
I20260812 19:13:26.020534 21320 raft_consensus.cc:1261] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260812 19:13:25.971372 23371 raft_consensus.cc:2726] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:26.022717 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:26.032737 21329 raft_consensus.cc:2379] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:25.980728 23720 raft_consensus.cc:2781] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:26.033128 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:25.763083 23754 consensus_queue.cc:237] T 784d71ae37454b3a9e8023f268428468 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:26.027113 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:26.055827 21318 raft_consensus.cc:2396] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260812 19:13:26.042285 21155 raft_consensus.cc:1261] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260812 19:13:26.044057 23749 log.cc:924] Time spent T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.536s user 0.002s sys 0.012s
W20260812 19:13:26.045554 23372 consensus_meta.cc:287] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.590s user 0.009s sys 0.000s
W20260812 19:13:26.047319 23772 log.cc:924] Time spent T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20260812 19:13:26.047211 23608 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:26.047844 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 252760us
I20260812 19:13:25.960436 23602 raft_consensus.cc:729] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:26.037643 21112 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:25.768277 23744 consensus_queue.cc:260] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:26.021286 23680 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:25.985280 23729 consensus_queue.cc:1035] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
I20260812 19:13:26.075969 21323 raft_consensus.cc:1226] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
W20260812 19:13:26.076463 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 161489us
I20260812 19:13:25.979980 23313 consensus_queue.cc:260] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:26.034669 23623 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:26.024181 23767 raft_consensus.cc:2781] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:26.094437 21318 raft_consensus.cc:1226] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:26.075079 22053 raft_consensus.cc:2396] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260812 19:13:26.105731 21328 raft_consensus.cc:1226] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
W20260812 19:13:26.113449 22051 consensus_meta.cc:287] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.671s user 0.014s sys 0.000s
I20260812 19:13:26.103711 21203 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:26.052726 23720 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:26.155355 23605 raft_consensus.cc:2781] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:26.155766 21204 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:26.155992 23605 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:26.162261 23605 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:26.158385 21323 raft_consensus.cc:1261] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260812 19:13:26.164872 23777 log.cc:924] Time spent T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
W20260812 19:13:26.171320 23761 log.cc:924] Time spent T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.431s user 0.001s sys 0.000s
I20260812 19:13:26.174283 23605 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:26.189376 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1457) took 2089 ms. Trace:
I20260812 19:13:26.195487 23619 raft_consensus.cc:3032] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
W20260812 19:13:26.203197 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 174855us
I20260812 19:13:26.207331 23313 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:26.209613 23720 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:26.213481 23602 consensus_queue.cc:260] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:26.216789 23697 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:26.218093 23744 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:26.163542 23769 log.cc:924] Time spent T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.246s user 0.001s sys 0.000s
I20260812 19:13:26.148849 22051 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:13:26.331802 21288 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
W20260812 19:13:26.285290 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 182047us
I20260812 19:13:26.288348 22053 raft_consensus.cc:1226] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260812 19:13:26.289812 21320 rpcz_store.cc:276] 0812 19:13:24.100207 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:24.104745 (+ 4538us) service_pool.cc:227] Handling call
0812 19:13:24.185002 (+ 80257us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:26.021061 (+1836059us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:26.189022 (+167961us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400084b54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:26.189287 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2610954752}
W20260812 19:13:26.294790 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 267210us
W20260812 19:13:26.283241 23206 log.cc:924] Time spent T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.061s user 0.000s sys 0.003s
I20260812 19:13:26.320282 21287 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706
W20260812 19:13:26.327322 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 171679us
I20260812 19:13:26.338997 21204 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:26.119511 23767 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:26.116374 23372 consensus_queue.cc:1035] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.184s
I20260812 19:13:26.300032 23651 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:26.296952 23619 raft_consensus.cc:729] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:26.108142 23680 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:26.074847 23713 consensus_queue.cc:260] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:26.406673 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 70938.8us
I20260812 19:13:26.308943 23735 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:26.159551 23750 consensus_queue.cc:260] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:26.456811 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1732) took 1218 ms. Trace:
I20260812 19:13:26.318446 23371 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260812 19:13:26.226033 23428 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:26.233184 23623 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:26.471192 21200 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.697s user 0.008s sys 0.046s
I20260812 19:13:26.345157 23697 raft_consensus.cc:2781] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election won for term 5
W20260812 19:13:26.471747 21200 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.697s user 0.008s sys 0.046s
W20260812 19:13:26.472404 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 699142us
W20260812 19:13:26.423208 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 69052.4us
W20260812 19:13:26.453137 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 114245us
I20260812 19:13:26.473199 21200 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:26.348304 23767 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:26.463088 21324 rpcz_store.cc:276] 0812 19:13:25.238221 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:25.238484 (+ 263us) service_pool.cc:227] Handling call
0812 19:13:25.266461 (+ 27977us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:25.306043 (+ 39582us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:25.306089 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:25.355243 (+ 49154us) log.cc:842] Serialized 26 byte log entry
0812 19:13:25.533800 (+178557us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:25.533861 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:25.533969 (+ 108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:26.456465 (+922496us) raft_consensus.cc:1671] finished
0812 19:13:26.456499 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:26.456725 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36038,"raft.queue_time_us":41941,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":12834,"thread_start_us":139414,"threads_started":1,"wal-append.queue_time_us":239031}
W20260812 19:13:26.493930 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 173800us
I20260812 19:13:26.331224 23758 consensus_queue.cc:1046] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.130s
I20260812 19:13:26.457659 23602 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:26.347621 23751 consensus_queue.cc:1035] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.142s
I20260812 19:13:26.508118 23206 tablet_bootstrap.cc:492] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260812 19:13:26.454664 23585 raft_consensus.cc:2726] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:26.460259 23605 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:26.498104 23697 raft_consensus.cc:686] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:26.434211 23432 consensus_queue.cc:1046] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
I20260812 19:13:26.438294 23681 raft_consensus.cc:2741] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20260812 19:13:26.507759 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 470179us
W20260812 19:13:26.568898 22008 leader_election.cc:334] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.602s (SENT)
I20260812 19:13:26.589272 22008 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:26.500185 23750 raft_consensus.cc:3037] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:26.575943 21287 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:26.592216 23658 raft_consensus.cc:2741] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260812 19:13:26.534217 21204 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:26.596330 23697 consensus_queue.cc:237] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:26.439788 23764 consensus_queue.cc:1035] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.094s
I20260812 19:13:26.577872 23371 consensus_queue.cc:1035] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260812 19:13:26.600029 23428 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:26.579031 23257 consensus_queue.cc:1035] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260812 19:13:26.436925 23770 consensus_queue.cc:1035] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.137s
I20260812 19:13:26.463382 23651 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:26.604954 23651 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:26.607571 23206 tablet_bootstrap.cc:492] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: Bootstrap complete.
I20260812 19:13:26.608758 23206 ts_tablet_manager.cc:1397] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: Time spent bootstrapping tablet: real 3.133s user 0.023s sys 0.032s
W20260812 19:13:26.633072 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 44347.2us
W20260812 19:13:26.692292 23722 consensus_meta.cc:287] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.132s user 0.016s sys 0.001s
W20260812 19:13:26.692479 23633 consensus_meta.cc:287] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.924s user 0.014s sys 0.000s
I20260812 19:13:26.745541 21114 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:26.740995 23774 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.015s sys 0.030s
I20260812 19:13:26.501410 23676 consensus_queue.cc:1035] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.292s
I20260812 19:13:26.846315 23775 consensus_queue.cc:1046] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.104s
I20260812 19:13:26.691880 23764 consensus_queue.cc:1046] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.153s
I20260812 19:13:26.800645 23658 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:26.841144 23774 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.687s user 0.015s sys 0.030s
I20260812 19:13:27.115398 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1487) took 1137 ms. Trace:
W20260812 19:13:26.871301 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 125911us
I20260812 19:13:26.906179 23789 raft_consensus.cc:2726] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:26.771670 23788 consensus_queue.cc:1035] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.366s
I20260812 19:13:27.112926 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:27.007014 23621 consensus_meta.cc:287] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.496s user 0.015s sys 0.001s
W20260812 19:13:27.116464 23608 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.005s sys 0.007s
W20260812 19:13:27.042852 23784 log.cc:924] Time spent T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.604s user 0.002s sys 0.000s
I20260812 19:13:26.804132 23722 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.129078 23658 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.130138 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1463) took 1359 ms. Trace:
I20260812 19:13:27.130811 23585 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:27.131999 23779 raft_consensus.cc:2726] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:13:27.132217 23585 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:27.132736 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 658608us
I20260812 19:13:27.128422 23621 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:27.130636 21327 rpcz_store.cc:276] 0812 19:13:25.770407 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:25.794161 (+ 23754us) service_pool.cc:227] Handling call
0812 19:13:25.929465 (+135304us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:25.952268 (+ 22803us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:25.952311 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:25.953495 (+ 1184us) log.cc:842] Serialized 26 byte log entry
0812 19:13:26.078150 (+124655us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:26.211207 (+133057us) spinlock_profiling.cc:243] Waited 6.84 ms on lock 0x7b48000fc0d8. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52de94d7f1 00007f52de9448e7 00007f52de94477a 00007f52dea3895f 00007f52dea38627 00007f52dea4cb2c 00007f52dea4cad8 00007f52dea4ca58 00007f52dea4ca13 00007f52dea4bcbc 00007f52dd9cdbe6
0812 19:13:26.540107 (+328900us) log.cc:842] Serialized 14 byte log entry
0812 19:13:27.104694 (+564587us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:27.104786 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:27.129801 (+ 25015us) raft_consensus.cc:1671] finished
0812 19:13:27.129923 (+ 122us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:27.130063 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51678,"prepare.queue_time_us":808193,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":29840,"raft.queue_time_us":51493,"raft.run_cpu_time_us":1355,"raft.run_wall_time_us":254071,"spinlock_wait_cycles":15730560,"thread_start_us":370067,"threads_started":2,"wal-append.queue_time_us":344267}
I20260812 19:13:27.131170 23782 consensus_queue.cc:1046] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
W20260812 19:13:26.869933 23313 consensus_meta.cc:287] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.570s user 0.010s sys 0.000s
W20260812 19:13:27.117193 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 538524us
I20260812 19:13:27.135828 23658 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:26.994938 23623 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:27.112214 23806 raft_consensus.cc:2726] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260812 19:13:27.123852 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 218236us
I20260812 19:13:27.149761 21200 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:27.138415 23608 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.558s user 0.006s sys 0.008s
W20260812 19:13:27.114766 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 538954us
W20260812 19:13:26.921583 21151 consensus_meta.cc:287] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.992s user 0.009s sys 0.005s
I20260812 19:13:27.175719 21151 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:27.125650 21112 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:27.139182 23764 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:27.005235 21322 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.007s sys 0.021s
W20260812 19:13:27.176819 21322 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.719s user 0.012s sys 0.023s
I20260812 19:13:27.150156 21114 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:27.171096 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:27.179148 22055 raft_consensus.cc:2396] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260812 19:13:27.181244 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260812 19:13:27.182411 21151 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51004 (request call id 234) took 1421 ms (client timeout 1524 ms). Trace:
I20260812 19:13:27.125162 21318 rpcz_store.cc:276] 0812 19:13:25.979833 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:26.119267 (+139434us) service_pool.cc:227] Handling call
0812 19:13:26.280383 (+161116us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:26.323459 (+ 43076us) raft_consensus.cc:1520] Early marking committed up to index 2
0812 19:13:26.843069 (+519610us) log.cc:842] Serialized 14 byte log entry
0812 19:13:27.115054 (+271985us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:27.115109 (+ 55us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:27.115155 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:27.115223 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:27.115340 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":787787,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":9058,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":145,"spinlock_wait_cycles":97664,"thread_start_us":628915,"threads_started":2,"wal-append.queue_time_us":242501}
I20260812 19:13:27.144209 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:27.169126 23788 consensus_queue.cc:1046] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260812 19:13:27.190802 23371 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:27.190989 22046 raft_consensus.cc:2454] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:13:27.191417 23371 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.180776 23806 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.192785 23806 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:26.803885 23633 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:27.193851 23371 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:27.186888 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 652684us
I20260812 19:13:27.187736 23764 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.188239 23782 consensus_queue.cc:1225] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer f741102bfe1441d1b453e5a838649059 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260812 19:13:27.187724 23612 raft_consensus.cc:2726] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260812 19:13:27.189909 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 50653.9us
I20260812 19:13:27.166301 23785 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.134063 23722 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:27.182945 21151 rpcz_store.cc:269] 0812 19:13:25.754759 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:25.754991 (+ 232us) service_pool.cc:227] Handling call
0812 19:13:27.176270 (+1421279us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:27.198175 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1488) took 1168 ms. Trace:
I20260812 19:13:27.185420 23775 consensus_queue.cc:1225] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer 63db10063a7d4eb2a9cc9dc952654706 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260812 19:13:27.194486 23371 raft_consensus.cc:2726] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260812 19:13:27.177663 23573 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.012s sys 0.010s
I20260812 19:13:27.226812 21287 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:27.228444 21200 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:27.149310 23621 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.238478 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:27.239281 21233 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
W20260812 19:13:27.240696 22008 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.017s sys 0.013s
W20260812 19:13:27.241294 22008 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.612s user 0.017s sys 0.013s
W20260812 19:13:27.242151 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 673271us
I20260812 19:13:27.242267 21254 tablet_copy_service.cc:342] P d1e1da0b9585460e8c365bf55679d4fa: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-18c9ea4c9e414d1093d28b6c91e19ff0 received from {username='slave'} at 127.0.0.1:57682
I20260812 19:13:27.222537 21328 rpcz_store.cc:276] 0812 19:13:26.029582 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:26.120751 (+ 91169us) service_pool.cc:227] Handling call
0812 19:13:26.282182 (+161431us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:26.282421 (+ 239us) raft_consensus.cc:1520] Early marking committed up to index 6
0812 19:13:26.781937 (+499516us) log.cc:842] Serialized 14 byte log entry
0812 19:13:27.197672 (+415735us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:27.197750 (+ 78us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:27.197815 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:27.197910 (+ 95us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:27.198091 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":748972,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":8209,"raft.run_cpu_time_us":63,"raft.run_wall_time_us":63,"spinlock_wait_cycles":19072,"thread_start_us":885869,"threads_started":3,"wal-append.queue_time_us":403064}
I20260812 19:13:27.248400 21200 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:27.233472 23573 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s user 0.013s sys 0.010s
W20260812 19:13:27.252405 23602 consensus_meta.cc:287] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.721s user 0.012s sys 0.000s
I20260812 19:13:27.253011 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:27.169337 23789 consensus_queue.cc:1035] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20260812 19:13:27.254927 21200 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:27.259891 23372 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:27.259959 23608 raft_consensus.cc:2726] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:27.260588 23372 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.261775 23619 consensus_queue.cc:260] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.262800 23791 raft_consensus.cc:2726] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:27.262969 21200 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:27.263265 23619 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:27.264094 23722 raft_consensus.cc:2781] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:27.253785 21241 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:13:27.264474 23722 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:27.256669 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:27.264920 23722 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:27.267650 23585 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:27.270231 23652 raft_consensus.cc:3032] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:27.272063 23652 raft_consensus.cc:729] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:27.272333 23257 raft_consensus.cc:3032] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:27.272151 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:27.272814 23257 raft_consensus.cc:729] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:27.273274 21238 raft_consensus.cc:2354] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 3. Current term is 4.
I20260812 19:13:27.273173 23652 consensus_queue.cc:260] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.273775 23257 consensus_queue.cc:260] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.274367 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:27.275034 21238 raft_consensus.cc:1261] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:13:27.230413 23823 raft_consensus.cc:2781] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:27.276446 21200 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:27.277043 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:27.311292 21233 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:27.237345 23774 raft_consensus.cc:2726] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:27.279650 21240 raft_consensus.cc:1226] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:27.280169 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:27.243389 23612 raft_consensus.cc:2726] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:13:27.308362 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:27.310381 21244 raft_consensus.cc:2396] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260812 19:13:27.314963 23585 raft_consensus.cc:2781] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:27.316694 23770 consensus_queue.cc:1035] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 19:13:27.318699 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:27.240969 23754 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.744s user 0.003s sys 0.022s
W20260812 19:13:27.276070 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 150525us
I20260812 19:13:27.246304 23658 raft_consensus.cc:2726] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:27.220265 23785 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.338007 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:27.338174 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:27.343365 21112 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
W20260812 19:13:27.338493 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 124272us
I20260812 19:13:27.347880 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1482) took 1382 ms. Trace:
I20260812 19:13:27.349989 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:27.351486 21318 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260812 19:13:27.351256 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:27.353417 21328 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
W20260812 19:13:27.363183 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 186876us
I20260812 19:13:27.363685 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:27.364790 21238 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:27.366283 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:27.356216 21322 rpcz_store.cc:276] 0812 19:13:25.965964 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:26.053422 (+ 87458us) service_pool.cc:227] Handling call
0812 19:13:26.092223 (+ 38801us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:26.108865 (+ 16642us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:26.108925 (+ 60us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:26.200787 (+ 91862us) spinlock_profiling.cc:243] Waited 29.2 ms on lock 0x7b48000fc0d8. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52de94d7f1 00007f52de9448e7 00007f52de94477a 00007f52dea3895f 00007f52dea38627 00007f52dea4cb2c 00007f52dea4cad8 00007f52dea4ca58 00007f52dea4ca13 00007f52dea4bcbc 00007f52dd9cdbe6
0812 19:13:26.318880 (+118093us) log.cc:842] Serialized 532 byte log entry
0812 19:13:27.177389 (+858509us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:27.177452 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:27.177549 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:27.347466 (+169917us) raft_consensus.cc:1671] finished
0812 19:13:27.347499 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:27.347796 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":170,"raft.run_cpu_time_us":1065,"raft.run_wall_time_us":84807,"spinlock_wait_cycles":67168512,"thread_start_us":719815,"threads_started":1,"wal-append.queue_time_us":922873}
W20260812 19:13:27.329110 23735 consensus_meta.cc:287] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.854s user 0.007s sys 0.004s
W20260812 19:13:27.394295 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 244205us
W20260812 19:13:27.342305 23754 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.845s user 0.003s sys 0.023s
I20260812 19:13:27.395500 21114 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:27.321990 23823 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:27.396377 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 53125.6us
I20260812 19:13:27.235977 23206 raft_consensus.cc:348] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.398478 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:27.404845 21328 raft_consensus.cc:2454] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:27.398885 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:27.397310 21288 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:13:27.338843 23608 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.343178 23821 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.321144 23676 consensus_queue.cc:1035] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.078s
I20260812 19:13:27.403549 21112 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:27.406916 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:27.413970 21327 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:27.399729 23697 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.007s sys 0.007s
I20260812 19:13:27.327486 21254 tablet_copy_service.cc:433] P d1e1da0b9585460e8c365bf55679d4fa: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-18c9ea4c9e414d1093d28b6c91e19ff0 on tablet 18c9ea4c9e414d1093d28b6c91e19ff0 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:27.403834 23791 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.399317 23735 raft_consensus.cc:2726] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:27.442310 23791 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.405543 23823 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:27.467136 23791 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:27.446971 23751 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:27.469223 23805 log.cc:924] Time spent T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.344s user 0.002s sys 0.001s
W20260812 19:13:27.451962 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 56745.3us
I20260812 19:13:27.436950 23821 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:27.471479 23802 log.cc:924] Time spent T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.345s user 0.002s sys 0.000s
W20260812 19:13:27.452425 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 48994.9us
I20260812 19:13:27.472999 23821 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:27.473248 21114 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:27.473671 21112 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:27.468979 23751 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.477227 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1479) took 1531 ms. Trace:
I20260812 19:13:27.477815 23751 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:27.401578 23372 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:27.478451 23428 raft_consensus.cc:3032] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:27.478781 23757 raft_consensus.cc:2726] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:27.452685 23825 raft_consensus.cc:2726] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:27.479835 23757 raft_consensus.cc:2726] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:27.480497 23713 raft_consensus.cc:3037] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:27.480834 23825 consensus_queue.cc:1046] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:13:27.481315 23757 consensus_queue.cc:1046] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:27.481869 23825 consensus_queue.cc:1035] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:13:27.453353 23573 raft_consensus.cc:2726] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260812 19:13:27.453485 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 53671.5us
I20260812 19:13:27.485525 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:27.485755 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:27.353542 23633 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:27.453847 23827 raft_consensus.cc:1050] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:13:27.428907 23608 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.488457 21157 raft_consensus.cc:1261] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260812 19:13:27.489323 23608 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:27.410854 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:27.417732 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:27.418774 22011 leader_election.cc:398] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 3. Current term is 4.
I20260812 19:13:27.497690 21243 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:27.428047 23621 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:27.428341 23807 log.cc:924] Time spent T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20260812 19:13:27.395828 23652 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:27.429839 22008 leader_election.cc:334] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.845s (SENT)
I20260812 19:13:27.506824 22008 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:27.397886 23774 raft_consensus.cc:2726] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:27.492497 23585 raft_consensus.cc:686] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:27.508669 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:27.509851 23585 consensus_queue.cc:237] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.510537 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:27.440606 23819 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.512079 23819 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.489813 23827 consensus_queue.cc:237] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.513922 21156 raft_consensus.cc:1226] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260812 19:13:27.514909 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:27.456854 23592 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.516283 23592 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.396173 23803 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.397979 23612 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.523581 23612 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.525029 23770 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.526017 23612 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:27.526243 23608 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.427441 23764 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:27.412784 23658 raft_consensus.cc:2781] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:27.526952 23658 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:27.527093 23680 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:27.527384 23658 raft_consensus.cc:3037] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:27.478924 23428 raft_consensus.cc:729] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:27.487350 21154 raft_consensus.cc:1226] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:27.411199 21287 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:27.409583 21324 raft_consensus.cc:1261] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260812 19:13:27.464426 23785 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:27.460356 23206 raft_consensus.cc:729] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Initialized, Role: LEARNER
I20260812 19:13:27.541443 21320 raft_consensus.cc:3032] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:27.541879 21320 raft_consensus.cc:729] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:27.518435 23592 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:27.498072 22011 leader_election.cc:401] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20260812 19:13:27.559751 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 141141us
I20260812 19:13:27.525604 23770 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.562003 23770 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:27.562951 23592 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:27.539548 23803 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.563429 21244 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:34764
W20260812 19:13:27.564062 23750 consensus_meta.cc:287] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.970s user 0.010s sys 0.002s
I20260812 19:13:27.564237 21244 raft_consensus.cc:595] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Received request to transfer leadership to TS f741102bfe1441d1b453e5a838649059
I20260812 19:13:27.564656 21244 raft_consensus.cc:603] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:13:27.565121 23803 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:27.565769 23803 consensus_queue.cc:1035] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:13:27.423226 21327 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:27.498164 21155 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
W20260812 19:13:27.439265 23697 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.560s user 0.007s sys 0.007s
I20260812 19:13:27.477662 21329 rpcz_store.cc:276] 0812 19:13:25.945718 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:26.034221 (+ 88503us) service_pool.cc:227] Handling call
0812 19:13:26.036725 (+ 2504us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:26.095086 (+ 58361us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:26.095133 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:26.095675 (+ 542us) log.cc:842] Serialized 26 byte log entry
0812 19:13:26.493911 (+398236us) spinlock_profiling.cc:243] Waited 30.7 ms on lock 0x7b180030acc0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b6c345 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b89e22 00007f52d97275bb 00007f52d972d063 00007f52d972d411 00007f52d9a7a5cf
0812 19:13:26.895227 (+401316us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:26.895288 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:26.895398 (+ 110us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:27.468523 (+573125us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:27.476990 (+ 8467us) raft_consensus.cc:1671] finished
0812 19:13:27.477013 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:27.477153 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":23028,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":19894,"spinlock_wait_cycles":98586112,"thread_start_us":400940,"threads_started":1,"wal-append.queue_time_us":1019141}
I20260812 19:13:27.505115 21233 raft_consensus.cc:2396] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260812 19:13:27.572319 21321 raft_consensus.cc:1226] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:27.503604 23757 raft_consensus.cc:3032] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:27.456288 23578 raft_consensus.cc:3032] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:27.426982 23806 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:27.544107 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:27.492975 21114 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:27.547369 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:27.584798 21241 raft_consensus.cc:2396] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260812 19:13:27.585148 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 92298us
I20260812 19:13:27.586508 21114 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:27.548172 23720 consensus_meta.cc:287] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.210s user 0.012s sys 0.000s
I20260812 19:13:27.587802 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:27.592057 21114 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:27.587802 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:27.548699 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:27.549094 23428 consensus_queue.cc:260] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.540434 23608 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:27.551323 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 152082us
I20260812 19:13:27.547376 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:27.551543 23680 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:27.550992 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:27.603068 21238 raft_consensus.cc:2396] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260812 19:13:27.551131 23821 raft_consensus.cc:2726] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:27.552071 23774 consensus_queue.cc:1035] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:13:27.533186 23651 raft_consensus.cc:2726] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:27.553499 23257 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:27.555675 23206 consensus_queue.cc:260] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:27.557158 21320 consensus_queue.cc:260] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.506312 22048 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:13:27.567902 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:27.639321 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:27.562307 22011 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
W20260812 19:13:27.519204 23767 consensus_meta.cc:287] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.009s user 0.005s sys 0.006s
W20260812 19:13:27.570056 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 140239us
W20260812 19:13:27.641628 22008 leader_election.cc:334] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.847s (SENT)
I20260812 19:13:27.514487 23819 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:27.573750 23658 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.582084 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:27.539803 23612 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.595484 21327 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:27.597361 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:27.500375 23722 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.632907 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:27.646306 21150 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:27.579289 23757 raft_consensus.cc:729] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:27.597937 23834 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.583024 23578 raft_consensus.cc:729] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:27.645228 21321 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:27.647524 22008 leader_election.cc:302] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:27.590281 23750 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:27.653473 21238 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "65db14a4838d40218947e8bd3420e892"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:13:27.662009 21238 raft_consensus.cc:595] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Received request to transfer leadership to TS f741102bfe1441d1b453e5a838649059
I20260812 19:13:27.662402 21238 raft_consensus.cc:603] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:13:27.646898 23623 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.657759 23658 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:27.633177 23432 raft_consensus.cc:2726] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:27.596737 23720 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.688210 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:27.670653 21320 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:13:27.703753 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 62217.9us
W20260812 19:13:27.687037 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 124796us
I20260812 19:13:27.670509 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:27.689109 21288 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:13:27.712479 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260812 19:13:27.647498 23803 raft_consensus.cc:2781] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:27.711381 21327 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:27.706013 21322 raft_consensus.cc:2454] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:27.704375 21200 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:27.714998 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 123091us
I20260812 19:13:27.659212 23825 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:27.668766 23764 raft_consensus.cc:3032] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:27.738279 22011 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
W20260812 19:13:27.748649 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 59622.9us
I20260812 19:13:27.723754 21321 raft_consensus.cc:1261] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260812 19:13:27.597780 23770 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.751497 23751 raft_consensus.cc:2781] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:27.751642 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260812 19:13:27.687942 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 46511.7us
I20260812 19:13:27.769122 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:27.687599 23428 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:27.670966 23834 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:27.752146 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 47912.8us
I20260812 19:13:27.753474 21114 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:27.794792 21200 leader_election.cc:302] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:27.760751 22011 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:27.646492 23680 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:27.777570 23608 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:27.797914 22011 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:27.692062 23750 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.754510 23764 raft_consensus.cc:729] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:27.805142 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:27.779138 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:27.780094 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:27.780884 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:27.806087 21322 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:27.761001 23592 raft_consensus.cc:2726] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 3. Current term is 4.
I20260812 19:13:27.766045 23819 raft_consensus.cc:2726] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:27.783700 23573 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.769697 23751 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:27.784379 23720 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:27.673900 23623 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.668336 23806 raft_consensus.cc:2726] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:27.792577 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:27.714727 23785 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20260812 19:13:27.778443 23425 raft_consensus.cc:2726] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:27.653473 23612 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.809646 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:27.765084 23803 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:27.869388 23825 raft_consensus.cc:686] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:27.815814 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:13:27.802326 23791 consensus_queue.cc:1035] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260812 19:13:27.819088 21287 leader_election.cc:398] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5.
I20260812 19:13:27.761129 23578 consensus_queue.cc:260] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.647389 23823 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.820662 22053 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEARNER]: Advancing to term 5
I20260812 19:13:27.760205 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:27.821069 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:27.829005 21320 raft_consensus.cc:1261] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260812 19:13:27.879880 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:27.831940 22055 raft_consensus.cc:1226] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:27.789228 23770 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.647811 23767 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.835976 21236 raft_consensus.cc:2454] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:27.754062 23757 consensus_queue.cc:260] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.837668 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:13:27.840893 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 132266us
W20260812 19:13:27.842566 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 47912.2us
I20260812 19:13:27.847339 21149 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:27.812328 23713 raft_consensus.cc:2781] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:27.808640 23651 raft_consensus.cc:2726] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:27.869660 23852 raft_consensus.cc:2726] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:27.814792 21323 raft_consensus.cc:1261] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260812 19:13:27.871975 23803 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:27.833871 23751 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:27.875804 21287 leader_election.cc:401] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:13:27.844885 23821 raft_consensus.cc:2781] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:27.785995 23722 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:27.878999 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 125658us
I20260812 19:13:27.878756 21150 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:27.800407 23371 consensus_queue.cc:1046] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260812 19:13:27.916595 21152 raft_consensus.cc:1226] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260812 19:13:27.860870 23720 raft_consensus.cc:2726] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:27.881270 23819 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:27.890141 23780 log.cc:924] Time spent T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 1.446s user 0.012s sys 0.001s
I20260812 19:13:27.840441 23573 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:27.911859 21114 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:27.888445 22048 raft_consensus.cc:3037] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:27.918674 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 99840.1us
I20260812 19:13:27.919636 21200 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:27.896487 23806 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.927449 22051 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:27.901604 23827 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.900008 23849 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.933212 23819 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.958101 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:27.941270 23612 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:27.935274 23713 raft_consensus.cc:686] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:27.945931 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:27.947149 21200 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:27.952777 21203 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:27.939280 23206 ts_tablet_manager.cc:1428] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: Time spent starting tablet: real 1.330s user 0.014s sys 0.002s
W20260812 19:13:27.964506 23206 ts_tablet_manager.cc:1460] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: Tablet startup took 1355ms
I20260812 19:13:27.955617 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:27.965353 22051 raft_consensus.cc:2396] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260812 19:13:27.965758 21150 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:27.932945 23658 raft_consensus.cc:2726] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5.
I20260812 19:13:27.941133 22052 raft_consensus.cc:1261] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260812 19:13:27.961227 21114 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:27.964129 21151 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:27.967260 22051 raft_consensus.cc:1261] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260812 19:13:27.964946 23206 ts_tablet_manager.cc:1462] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: Trace:
0812 19:13:05.627040 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:05.628708 (+ 1668us) service_pool.cc:227] Handling call
0812 19:13:06.481020 (+852312us) ts_tablet_manager.cc:928] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: Initiating tablet copy from peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551)
0812 19:13:15.773126 (+9292106us) tablet_copy_client.cc:361] Tablet copy session begun
0812 19:13:17.880142 (+2107016us) tablet_metadata.cc:641] Metadata flushed
0812 19:13:17.918299 (+ 38157us) tablet_copy_client.cc:490] Wrote new tablet metadata
0812 19:13:18.384709 (+466410us) tablet_copy_client.cc:411] Wrote new consensus metadata
0812 19:13:18.385049 (+ 340us) tablet_replica.cc:205] Creating consensus instance
0812 19:13:18.385418 (+ 369us) ts_tablet_manager.cc:983] Replica registered.
0812 19:13:18.385535 (+ 117us) inbound_call.cc:173] Queueing success response
0812 19:13:23.475677 (+5090142us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0812 19:13:26.609174 (+3133497us) ts_tablet_manager.cc:1429] Starting tablet replica
0812 19:13:27.102058 (+492884us) tablet_replica.cc:259] Starting instrumentation
0812 19:13:27.102894 (+ 836us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":30768,"mutex_wait_us":1591,"raft.queue_time_us":15400,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":17092,"tablet-copy.queue_time_us":358149,"tablet-download-pool.queue_time_us":2066,"tablet-download-pool.run_cpu_time_us":1709,"tablet-download-pool.run_wall_time_us":901964,"thread_start_us":1574536,"threads_started":2}
I20260812 19:13:27.976059 21151 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:27.980013 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:27.982571 21200 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:27.944449 23720 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:27.986505 21203 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:27.980839 21157 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:27.894579 23764 consensus_queue.cc:260] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:28.006379 21154 raft_consensus.cc:1226] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260812 19:13:28.009156 23842 log.cc:924] Time spent T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.160s user 0.001s sys 0.001s
I20260812 19:13:27.962016 23849 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:28.013306 23849 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:27.963692 23774 raft_consensus.cc:2781] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election won for term 4
W20260812 19:13:28.015053 23851 log.cc:924] Time spent T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.151s user 0.001s sys 0.000s
W20260812 19:13:28.015565 23813 log.cc:924] Time spent T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.782s user 0.002s sys 0.000s
I20260812 19:13:28.016371 23612 raft_consensus.cc:2726] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:28.016543 23834 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:28.018493 23823 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:28.018710 23850 consensus_queue.cc:1046] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260812 19:13:28.019764 23850 consensus_queue.cc:1225] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [LEADER]: Peer d1e1da0b9585460e8c365bf55679d4fa log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260812 19:13:28.020167 23767 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:28.021267 23713 consensus_queue.cc:237] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.935309 23825 consensus_queue.cc:237] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:27.954669 23676 raft_consensus.cc:2726] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:28.020345 23850 raft_consensus.cc:3032] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:28.177286 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:28.041291 22053 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260812 19:13:28.027024 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:28.183772 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 201270us
W20260812 19:13:28.186697 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 149639us
I20260812 19:13:28.180207 22056 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40614 (request call id 419) took 2446 ms. Trace:
I20260812 19:13:28.187270 21320 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 6
I20260812 19:13:28.126642 21319 raft_consensus.cc:1226] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
W20260812 19:13:28.172176 22008 leader_election.cc:334] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.636s (SENT)
I20260812 19:13:28.191246 22008 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:28.193076 21200 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:28.189940 22056 rpcz_store.cc:276] 0812 19:13:25.734155 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:25.734392 (+ 237us) service_pool.cc:227] Handling call
0812 19:13:25.734932 (+ 540us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:25.735503 (+ 571us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:25.735541 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:25.736140 (+ 599us) log.cc:842] Serialized 26 byte log entry
0812 19:13:25.866012 (+129872us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:25.866079 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:25.866744 (+ 665us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:27.702564 (+1835820us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:28.177914 (+475350us) raft_consensus.cc:1671] finished
0812 19:13:28.177950 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:28.180085 (+ 2135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1538,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":10098,"spinlock_wait_cycles":67890048,"thread_start_us":129168,"threads_started":1,"wal-append.queue_time_us":559436}
I20260812 19:13:28.196444 23313 raft_consensus.cc:2726] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:28.196708 23775 raft_consensus.cc:2781] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:28.043092 23371 raft_consensus.cc:2781] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:28.197304 23775 raft_consensus.cc:686] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:28.197570 23371 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:28.198065 23371 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:28.198132 22011 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:28.198415 23633 consensus_queue.cc:1035] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:13:28.199054 23608 consensus_queue.cc:1046] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 19:13:28.199602 22011 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:28.199925 23608 consensus_queue.cc:1225] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer 93578836968649f6bfcaecc709871412 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260812 19:13:28.200912 23633 raft_consensus.cc:3032] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:28.201360 23633 raft_consensus.cc:729] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:28.200768 23727 raft_consensus.cc:2781] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:28.198415 23775 consensus_queue.cc:237] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:28.025614 23735 raft_consensus.cc:2781] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:28.202354 23633 consensus_queue.cc:260] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:28.203033 23735 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:28.204180 23735 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:28.204790 23633 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:28.207104 21112 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:28.213294 23823 raft_consensus.cc:2781] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:28.213795 23823 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:28.214188 23823 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:28.219655 23841 log.cc:924] Time spent T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.260s user 0.000s sys 0.001s
I20260812 19:13:28.221897 23681 raft_consensus.cc:1013] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 3 LEADER]: attempt to promote peer 63db10063a7d4eb2a9cc9dc952654706: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260812 19:13:28.206768 23765 consensus_queue.cc:1035] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:28.207804 23767 consensus_queue.cc:1035] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 19:13:28.208636 23605 consensus_queue.cc:1035] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 19:13:28.223212 21200 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:28.226758 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:28.230376 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:28.231877 21322 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:28.231662 21200 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:28.234445 21112 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:28.234824 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:28.235095 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:28.235342 23849 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:28.235770 23849 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:28.237167 21200 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:28.093051 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 106605us
I20260812 19:13:28.189363 23623 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:28.166478 23774 raft_consensus.cc:686] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:28.241348 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:28.243582 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:28.247467 21203 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:28.237749 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:28.155407 21318 raft_consensus.cc:1261] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260812 19:13:28.127833 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 136915us
I20260812 19:13:27.979357 23592 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:28.250558 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 289426us
I20260812 19:13:28.254748 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:28.260098 21320 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:28.271471 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:28.268733 22052 raft_consensus.cc:2454] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:28.277345 21243 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:28.229544 23744 raft_consensus.cc:2781] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:28.276738 21233 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:28.167479 23680 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:28.229409 23727 raft_consensus.cc:686] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:28.236079 23791 raft_consensus.cc:2781] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:28.250371 23849 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:28.249410 23612 raft_consensus.cc:2781] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:28.128952 23832 raft_consensus.cc:2781] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:28.299953 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:27.963382 23806 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:28.278196 23744 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:28.310067 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 62721.7us
I20260812 19:13:28.155961 23750 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:28.026939 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:28.284976 21319 raft_consensus.cc:2932] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:13:28.251590 23850 raft_consensus.cc:729] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:28.312798 21112 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:28.301913 21114 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:27.961388 23827 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:28.333426 23834 raft_consensus.cc:2781] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:28.355054 23834 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:28.355438 23834 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:28.355513 23827 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:28.314746 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:28.356585 21157 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:28.279841 23775 raft_consensus.cc:2726] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:28.103070 23856 raft_consensus.cc:2741] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260812 19:13:27.990994 23836 consensus_queue.cc:1035] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.312s
I20260812 19:13:28.348896 21203 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:28.302047 23791 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:28.349226 23744 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:28.357942 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 98743us
W20260812 19:13:28.360689 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 48044.8us
I20260812 19:13:28.313905 23612 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:28.413391 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 176319us
W20260812 19:13:28.368949 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 74560.5us
W20260812 19:13:28.347982 23805 log.cc:924] Time spent T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
W20260812 19:13:28.369876 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 68125.5us
I20260812 19:13:28.272850 23758 consensus_queue.cc:1035] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260812 19:13:28.044112 23720 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:28.263571 23803 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:28.371884 23257 consensus_meta.cc:287] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.729s user 0.011s sys 0.002s
I20260812 19:13:28.363957 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:28.352125 23652 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:28.290181 23592 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:28.303489 23676 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260812 19:13:28.409943 22011 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:28.410821 21203 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:28.411208 21204 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:28.351858 23832 raft_consensus.cc:686] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:28.412541 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:28.373477 23573 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:28.516963 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 63360us
I20260812 19:13:28.582523 21238 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:28.582840 21200 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:28.516906 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:28.583995 21321 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
W20260812 19:13:28.571406 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 100419us
I20260812 19:13:28.388849 23765 raft_consensus.cc:2726] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:28.442843 23612 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:28.589486 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 178373us
I20260812 19:13:28.465126 23775 raft_consensus.cc:2781] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election won for term 6
W20260812 19:13:28.589421 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 178678us
I20260812 19:13:28.357815 23855 consensus_queue.cc:1035] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.237s
I20260812 19:13:28.374501 23827 raft_consensus.cc:2781] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election won for term 6
W20260812 19:13:28.574082 22048 consensus_meta.cc:287] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.611s user 0.012s sys 0.012s
W20260812 19:13:28.572352 23776 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.042s sys 0.072s
I20260812 19:13:28.389333 23791 raft_consensus.cc:3037] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:28.569552 23602 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:28.731127 23602 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:28.731350 23619 raft_consensus.cc:2932] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:13:28.639390 21239 raft_consensus.cc:2396] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260812 19:13:28.599738 23801 log.cc:924] Time spent T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.129s user 0.000s sys 0.001s
I20260812 19:13:28.447057 23371 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:28.736632 23821 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:28.632695 22046 raft_consensus.cc:1226] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:28.737317 23821 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:28.640913 23257 raft_consensus.cc:2781] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260812 19:13:28.738399 23870 log.cc:924] Time spent T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
I20260812 19:13:28.738710 23257 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:28.739255 23257 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:28.741752 23608 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:28.582165 23652 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:28.641546 23775 raft_consensus.cc:686] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:28.669529 23713 raft_consensus.cc:2781] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:28.677711 23750 raft_consensus.cc:2781] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:28.713313 22048 raft_consensus.cc:2454] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 4.
I20260812 19:13:28.720465 23827 raft_consensus.cc:686] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:28.720777 23803 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:28.451756 23727 consensus_queue.cc:237] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:28.722991 21169 tablet_copy_service.cc:342] P f741102bfe1441d1b453e5a838649059: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-c31c6a2accba4d29ba096500b9d1a911 received from {username='slave'} at 127.0.0.1:51004
I20260812 19:13:28.589449 23850 consensus_queue.cc:260] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:28.737023 23869 log.cc:924] Time spent T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
I20260812 19:13:28.414170 23774 consensus_queue.cc:237] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:28.742400 23608 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:28.717473 23776 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.693s user 0.043s sys 0.072s
I20260812 19:13:28.785211 22048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40614 (request call id 427) took 1096 ms. Trace:
I20260812 19:13:28.769021 23750 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:28.718798 23676 raft_consensus.cc:2781] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:28.790505 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:28.771483 23602 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:28.792507 21169 tablet_copy_service.cc:433] P f741102bfe1441d1b453e5a838649059: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-c31c6a2accba4d29ba096500b9d1a911 on tablet c31c6a2accba4d29ba096500b9d1a911 with peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:28.791860 21112 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:28.737347 23371 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:28.758275 23713 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:28.769799 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 359897us
I20260812 19:13:28.766881 23832 consensus_queue.cc:237] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:28.822513 23850 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:29.136502 23750 raft_consensus.cc:3037] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:29.174963 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:29.044344 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:29.139412 22048 rpcz_store.cc:276] 0812 19:13:27.688707 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:27.702379 (+ 13672us) service_pool.cc:227] Handling call
0812 19:13:28.785123 (+1082744us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:28.953732 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 517684us
W20260812 19:13:29.138104 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 346334us
I20260812 19:13:29.139019 23713 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:28.953246 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 370482us
W20260812 19:13:28.834380 23857 log.cc:924] Time spent T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.294s user 0.000s sys 0.001s
W20260812 19:13:29.077921 23735 consensus_meta.cc:287] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.844s user 0.006s sys 0.006s
W20260812 19:13:29.172041 23823 consensus_meta.cc:287] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.957s user 0.010s sys 0.002s
I20260812 19:13:28.821954 23806 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:29.031464 23859 log.cc:924] Time spent T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.222s user 0.001s sys 0.001s
I20260812 19:13:29.140380 23602 raft_consensus.cc:2781] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:29.195644 23602 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:29.196156 23602 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:29.197325 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1493) took 1465 ms. Trace:
I20260812 19:13:29.197613 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:29.197813 21327 rpcz_store.cc:276] 0812 19:13:27.731457 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:27.773765 (+ 42308us) service_pool.cc:227] Handling call
0812 19:13:27.811596 (+ 37831us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:27.863276 (+ 51680us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:27.863316 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:27.864056 (+ 740us) log.cc:842] Serialized 26 byte log entry
0812 19:13:28.237861 (+373805us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:28.237917 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:28.238009 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:29.197054 (+959045us) raft_consensus.cc:1671] finished
0812 19:13:29.197086 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:29.197254 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":20514,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":11193,"thread_start_us":396429,"threads_started":2,"wal-append.queue_time_us":468216}
I20260812 19:13:29.008693 23658 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:28.803459 21287 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:28.958878 21237 raft_consensus.cc:1226] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: [1.2-3.3]
I20260812 19:13:29.067462 21235 raft_consensus.cc:1226] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:29.046784 23827 consensus_queue.cc:237] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:29.153512 23371 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:29.153512 23608 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:29.156546 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:29.214099 21154 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:29.158491 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:29.159627 22011 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
W20260812 19:13:29.160066 23873 log.cc:924] Time spent T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.370s user 0.001s sys 0.000s
I20260812 19:13:29.216724 23806 raft_consensus.cc:2726] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260812 19:13:29.160561 23861 log.cc:924] Time spent T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.601s user 0.002s sys 0.000s
W20260812 19:13:29.216989 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 57534.2us
I20260812 19:13:29.218190 22011 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:29.163774 23652 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:29.219830 23806 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:28.934481 23775 consensus_queue.cc:237] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:28.883852 23866 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:29.220352 22055 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 605) took 1332 ms. Trace:
I20260812 19:13:29.221019 23866 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:29.137179 23676 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:29.223735 23676 raft_consensus.cc:3037] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:29.224745 23866 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:29.170889 21319 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:29.175760 21241 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
W20260812 19:13:29.201740 21114 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.020s sys 0.047s
I20260812 19:13:29.209990 21288 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:13:29.210305 22056 pending_rounds.cc:76] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706: Aborting all ops after (but not including) 5
I20260812 19:13:29.227351 22056 pending_rounds.cc:97] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260812 19:13:29.067946 23720 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:29.227708 22056 raft_consensus.cc:2866] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260812 19:13:29.226994 21114 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.653s user 0.020s sys 0.047s
I20260812 19:13:29.220824 22055 rpcz_store.cc:276] 0812 19:13:27.887826 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:27.896517 (+ 8691us) service_pool.cc:227] Handling call
0812 19:13:27.896921 (+ 404us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:27.898918 (+ 1997us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:27.898949 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:27.919094 (+ 20145us) log.cc:842] Serialized 26 byte log entry
0812 19:13:28.281901 (+362807us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:28.281959 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:28.282062 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:29.220105 (+938043us) raft_consensus.cc:1671] finished
0812 19:13:29.220126 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:29.220280 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2274,"raft.queue_time_us":7214,"raft.run_cpu_time_us":325,"raft.run_wall_time_us":23753,"thread_start_us":339934,"threads_started":1,"wal-append.queue_time_us":380809}
I20260812 19:13:29.229527 23803 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:29.230209 23803 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:29.185918 23860 log.cc:924] Time spent T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.591s user 0.001s sys 0.001s
I20260812 19:13:29.237367 22011 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:29.210502 23681 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:29.243359 23681 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:29.199993 23633 consensus_meta.cc:287] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.983s user 0.013s sys 0.006s
W20260812 19:13:29.246820 21114 connection.cc:232] RPC call timeout handler was delayed by -1.4762623050482944s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19030000000000002s for a response
I20260812 19:13:29.198746 22047 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:29.232872 23803 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:29.272509 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 335185us
I20260812 19:13:29.200372 22045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 606) took 1284 ms. Trace:
W20260812 19:13:29.319643 22008 leader_election.cc:334] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.703s (SENT)
I20260812 19:13:29.327396 22008 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:29.205399 23868 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:29.220350 23806 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:29.207484 23823 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:29.199719 23735 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:29.234306 23658 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:29.979858 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 660231us
W20260812 19:13:29.535820 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 327105us
I20260812 19:13:29.239464 23371 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:29.582353 23428 consensus_meta.cc:287] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.794s user 0.011s sys 0.006s
I20260812 19:13:29.942847 23803 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:29.732007 23872 log.cc:924] Time spent T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.557s user 0.001s sys 0.000s
I20260812 19:13:29.988817 23371 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:29.402339 22011 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 1.526s (SENT) [suppressed 21 similar messages]
I20260812 19:13:29.323072 22045 rpcz_store.cc:276] 0812 19:13:27.905063 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:27.909821 (+ 4758us) service_pool.cc:227] Handling call
0812 19:13:27.934698 (+ 24877us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:27.953674 (+ 18976us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:27.956702 (+ 3028us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:28.004168 (+ 47466us) log.cc:842] Serialized 26 byte log entry
0812 19:13:28.353426 (+349258us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:28.353541 (+ 115us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:28.353687 (+ 146us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:29.174399 (+820712us) raft_consensus.cc:1671] finished
0812 19:13:29.174419 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:29.189502 (+ 15083us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2608,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":10797,"thread_start_us":94791,"threads_started":1,"wal-append.queue_time_us":368782}
W20260812 19:13:29.561928 21157 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.030s sys 0.063s
W20260812 19:13:30.091439 21157 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.048s user 0.047s sys 0.129s
I20260812 19:13:29.462538 23868 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:29.991809 23806 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:29.339322 23720 consensus_queue.cc:1035] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.674s
I20260812 19:13:29.245451 23651 raft_consensus.cc:2781] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260812 19:13:29.865463 21115 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.013s sys 0.015s
W20260812 19:13:30.136297 21115 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.887s user 0.013s sys 0.017s
W20260812 19:13:29.562361 23834 consensus_meta.cc:287] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.206s user 0.015s sys 0.000s
W20260812 19:13:29.993234 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 752539us
W20260812 19:13:29.994663 21236 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57682 (request call id 207) took 2449 ms (client timeout 1526 ms). Trace:
W20260812 19:13:29.268903 23791 consensus_meta.cc:287] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.541s user 0.011s sys 0.001s
I20260812 19:13:29.980352 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1512) took 1568 ms. Trace:
W20260812 19:13:30.141079 21236 rpcz_store.cc:269] 0812 19:13:27.545121 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:27.545288 (+ 167us) service_pool.cc:227] Handling call
0812 19:13:29.994419 (+2449131us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b54002f17d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:29.994529 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":554054528}
I20260812 19:13:30.141309 21320 rpcz_store.cc:276] 0812 19:13:28.411644 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:28.412648 (+ 1004us) service_pool.cc:227] Handling call
0812 19:13:28.413016 (+ 368us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:28.468195 (+ 55179us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:28.468256 (+ 61us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:28.468965 (+ 709us) log.cc:842] Serialized 26 byte log entry
0812 19:13:28.954091 (+485126us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:28.954149 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:28.954254 (+ 105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:29.729096 (+774842us) raft_consensus.cc:1671] finished
0812 19:13:29.729127 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:29.980274 (+251147us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1600,"spinlock_wait_cycles":101137792,"thread_start_us":319183,"threads_started":1,"wal-append.queue_time_us":993728}
I20260812 19:13:30.002414 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:29.944195 21288 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.677s user 0.018s sys 0.028s
I20260812 19:13:30.003207 23681 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:30.004339 23845 log.cc:924] Time spent T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 2.095s user 0.023s sys 0.035s
I20260812 19:13:30.007849 23884 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:30.008666 23608 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:30.009204 23751 consensus_meta.cc:287] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 2.079s user 0.017s sys 0.000s
I20260812 19:13:30.010957 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:30.012706 23676 consensus_meta.cc:287] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.788s user 0.012s sys 0.001s
I20260812 19:13:29.419350 23762 raft_consensus.cc:2726] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:29.543169 23633 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:29.956931 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 1.18062e+06us
W20260812 19:13:29.729966 23878 log.cc:924] Time spent T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260812 19:13:29.697634 23775 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:30.055469 21240 raft_consensus.cc:1226] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
W20260812 19:13:30.018896 23849 consensus_meta.cc:287] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.686s user 0.015s sys 0.000s
W20260812 19:13:30.108017 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 705656us
I20260812 19:13:30.139622 21157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 243) took 1723 ms. Trace:
I20260812 19:13:30.145795 21235 raft_consensus.cc:2379] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:30.141336 21112 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:30.144507 21288 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.877s user 0.018s sys 0.030s
W20260812 19:13:30.175007 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 965026us
I20260812 19:13:30.062712 23803 raft_consensus.cc:504] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:30.092280 23658 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:30.127305 23833 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.341s user 0.004s sys 0.019s
I20260812 19:13:30.157780 21233 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:13:30.138350 23257 consensus_meta.cc:287] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.399s user 0.018s sys 0.000s
W20260812 19:13:30.159170 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 166229us
W20260812 19:13:30.142546 23821 consensus_meta.cc:287] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 1.405s user 0.017s sys 0.000s
I20260812 19:13:30.135946 23651 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:30.115602 23899 raft_consensus.cc:2726] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:30.172700 21157 rpcz_store.cc:276] 0812 19:13:28.407833 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:28.409292 (+ 1459us) service_pool.cc:227] Handling call
0812 19:13:28.581046 (+171754us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:28.591828 (+ 10782us) raft_consensus.cc:1520] Early marking committed up to index 6
0812 19:13:29.043513 (+451685us) log.cc:842] Serialized 14 byte log entry
0812 19:13:30.130776 (+1087263us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:30.130839 (+ 63us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:30.130904 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:30.130973 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:30.131106 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38651,"prepare.queue_time_us":688614,"prepare.run_cpu_time_us":3224,"prepare.run_wall_time_us":71808,"raft.queue_time_us":42406,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":9686,"thread_start_us":1499336,"threads_started":2}
I20260812 19:13:30.264705 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:30.148250 21286 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.008s sys 0.016s
W20260812 19:13:30.209714 21203 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.769s user 0.004s sys 0.017s
I20260812 19:13:30.264837 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:30.265897 21203 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.826s user 0.005s sys 0.017s
I20260812 19:13:30.189733 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:30.145424 23834 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:30.226879 22008 leader_election.cc:334] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.756s (SENT)
I20260812 19:13:30.255748 21287 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:30.211894 23775 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:30.160583 23898 raft_consensus.cc:2781] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:30.264119 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.265743 21318 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 5
W20260812 19:13:30.266296 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 826309us
I20260812 19:13:30.160238 23751 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:30.218865 23833 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.430s user 0.004s sys 0.019s
I20260812 19:13:30.196780 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:30.266580 21321 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 6
I20260812 19:13:30.186292 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:30.177870 23762 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:30.305884 21328 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:30.307507 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:30.267232 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:30.268390 23900 log.cc:924] Time spent T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.131s user 0.000s sys 0.000s
I20260812 19:13:30.319515 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:30.269503 23803 leader_election.cc:288] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.320222 23834 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.320683 22048 raft_consensus.cc:2379] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 4: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:30.321455 23868 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.321897 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:30.277432 23257 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:30.322866 21151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 247) took 1088 ms. Trace:
I20260812 19:13:30.271085 23911 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:30.271582 23619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:50988 (request call id 9) took 26712 ms. Trace:
I20260812 19:13:30.271785 23821 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:30.264070 23651 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:30.273707 23852 raft_consensus.cc:2781] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:30.275698 21237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1506) took 1029 ms. Trace:
I20260812 19:13:30.151793 23791 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:30.218680 23658 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:30.266314 21327 raft_consensus.cc:2454] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
W20260812 19:13:30.279285 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 52469.8us
I20260812 19:13:30.168537 23633 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:30.301247 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 115660us
I20260812 19:13:30.286402 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:30.285630 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:30.340862 21237 rpcz_store.cc:276] 0812 19:13:29.246312 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:29.246856 (+ 544us) service_pool.cc:227] Handling call
0812 19:13:29.247242 (+ 386us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:29.367806 (+120564us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:29.402351 (+ 34545us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:29.482933 (+ 80582us) log.cc:842] Serialized 26 byte log entry
0812 19:13:30.131047 (+648114us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:30.131080 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:30.131145 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:30.275331 (+144186us) raft_consensus.cc:1671] finished
0812 19:13:30.275363 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:30.275604 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":69098,"raft.queue_time_us":135220,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":54397,"thread_start_us":428743,"threads_started":1,"wal-append.queue_time_us":622005}
I20260812 19:13:30.350934 21322 raft_consensus.cc:2454] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
W20260812 19:13:30.285045 21204 connection.cc:232] RPC call timeout handler was delayed by -1.4007530710659921s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1713s for a response
I20260812 19:13:30.166774 23676 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:30.311869 21243 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.144s user 0.016s sys 0.031s
W20260812 19:13:30.354856 21243 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.187s user 0.017s sys 0.031s
I20260812 19:13:30.162103 23608 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:30.312343 23759 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:30.316550 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.318713 23898 raft_consensus.cc:686] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 4 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:30.318441 23762 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:30.323294 21151 rpcz_store.cc:276] 0812 19:13:29.234692 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:29.235508 (+ 816us) service_pool.cc:227] Handling call
0812 19:13:29.236057 (+ 549us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:29.236383 (+ 326us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:29.236423 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:29.277334 (+ 40911us) log.cc:842] Serialized 26 byte log entry
0812 19:13:29.663108 (+385774us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:29.663146 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:29.663250 (+ 104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:30.322642 (+659392us) raft_consensus.cc:1671] finished
0812 19:13:30.322663 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:30.322795 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":385400,"threads_started":1,"wal-append.queue_time_us":820035}
I20260812 19:13:30.399096 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:30.326462 21286 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.804s user 0.008s sys 0.016s
I20260812 19:13:30.162369 23884 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:30.337872 21241 raft_consensus.cc:1226] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:30.339183 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1753) took 2751 ms. Trace:
W20260812 19:13:30.359797 23602 consensus_meta.cc:287] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.163s user 0.013s sys 0.001s
I20260812 19:13:30.304155 23832 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:30.184404 23849 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:30.422098 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1497) took 1466 ms. Trace:
I20260812 19:13:30.310874 23856 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:30.304729 23855 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:30.349185 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.431581 21236 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:30.353302 21200 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:30.314951 23735 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.361449 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.437944 21237 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:30.439126 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:30.369961 21318 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:30.371246 23608 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:30.371729 23744 consensus_meta.cc:287] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.901s user 0.011s sys 0.004s
I20260812 19:13:30.441111 23744 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:30.444053 23744 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:30.445663 22055 raft_consensus.cc:2379] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:30.371826 23751 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.372630 23257 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.373514 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:30.448715 21322 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:30.349825 23681 raft_consensus.cc:2726] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:30.381036 22050 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEARNER]: Advancing to term 3
I20260812 19:13:30.384899 23578 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:30.386826 23907 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
W20260812 19:13:30.387972 21238 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57682 (request call id 208) took 2162 ms (client timeout 1671 ms). Trace:
W20260812 19:13:30.464116 21238 rpcz_store.cc:269] 0812 19:13:28.225251 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:28.225412 (+ 161us) service_pool.cc:227] Handling call
0812 19:13:30.387711 (+2162299us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b54001509d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:30.387886 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":797299584}
I20260812 19:13:30.390314 23791 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.391542 23605 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:30.392532 23896 log.cc:924] Time spent T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.136s user 0.000s sys 0.000s
I20260812 19:13:30.467288 23633 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:30.393040 23901 log.cc:924] Time spent T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20260812 19:13:30.468194 23917 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:30.469002 23917 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:30.469825 23883 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:30.470523 23883 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:30.469012 23898 consensus_queue.cc:237] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:30.473301 23883 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.392971 23759 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:30.476420 23917 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.394517 21287 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:30.477596 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 83187.6us
I20260812 19:13:30.478406 21287 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:30.479876 23902 raft_consensus.cc:2781] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:30.480618 23902 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:30.481616 23902 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:30.481367 23605 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:30.340517 23852 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:30.482860 23852 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:30.484027 23821 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:30.484674 23912 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:30.402104 22052 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 422) took 2023 ms. Trace:
I20260812 19:13:30.335757 22045 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 4.
I20260812 19:13:30.485354 23912 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:30.486852 22045 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEARNER]: Advancing to term 4
I20260812 19:13:30.488106 23912 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.405179 21204 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:30.400497 21235 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:30.340442 21233 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:30.490149 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.496356 21233 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:30.345791 23592 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.485356 22052 rpcz_store.cc:276] 0812 19:13:28.378530 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:28.378823 (+ 293us) service_pool.cc:227] Handling call
0812 19:13:28.413953 (+ 35130us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:28.590100 (+176147us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:28.590177 (+ 77us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:28.591682 (+ 1505us) log.cc:842] Serialized 52 byte log entry
0812 19:13:29.008539 (+416857us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:29.008599 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:29.008715 (+ 116us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:30.401799 (+1393084us) raft_consensus.cc:1671] finished
0812 19:13:30.401825 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:30.402025 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31912,"raft.queue_time_us":29865,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":34543,"thread_start_us":209173,"threads_started":1,"wal-append.queue_time_us":741824}
I20260812 19:13:30.412362 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:30.368808 22048 raft_consensus.cc:2396] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260812 19:13:30.391613 23904 log.cc:924] Time spent T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20260812 19:13:30.358970 23868 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:30.422775 21243 rpcz_store.cc:276] 0812 19:13:28.955569 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:28.955709 (+ 140us) service_pool.cc:227] Handling call
0812 19:13:29.150118 (+194409us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:29.150363 (+ 245us) raft_consensus.cc:1520] Early marking committed up to index 2
0812 19:13:30.355283 (+1204920us) log.cc:842] Serialized 14 byte log entry
0812 19:13:30.411199 (+ 55916us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:30.411265 (+ 66us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:30.421677 (+ 10412us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:30.421799 (+ 122us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:30.422003 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":35316,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":56,"raft.queue_time_us":88,"thread_start_us":1229275,"threads_started":2}
I20260812 19:13:30.360015 23833 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:30.460212 21114 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:30.461144 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 108032us
W20260812 19:13:30.525137 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 120094us
I20260812 19:13:30.525828 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.407644 23823 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.464768 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:30.471038 22056 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 424) took 1290 ms. Trace:
I20260812 19:13:30.527143 22056 rpcz_store.cc:276] 0812 19:13:29.180670 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:29.181147 (+ 477us) service_pool.cc:227] Handling call
0812 19:13:29.210108 (+ 28961us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:29.228558 (+ 18450us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:29.228596 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:29.229063 (+ 467us) log.cc:842] Serialized 26 byte log entry
0812 19:13:29.484565 (+255502us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:29.484652 (+ 87us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:29.484839 (+ 187us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:30.470710 (+985871us) raft_consensus.cc:1671] finished
0812 19:13:30.470739 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:30.470948 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":254736,"threads_started":1,"wal-append.queue_time_us":911180}
I20260812 19:13:30.393271 23788 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:30.476671 23759 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:30.395977 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:30.338052 23911 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:30.354419 23658 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:30.493635 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 143409us
I20260812 19:13:30.502427 21288 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
W20260812 19:13:30.512878 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 177357us
I20260812 19:13:30.494330 23735 raft_consensus.cc:2781] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:30.545845 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.520829 21237 raft_consensus.cc:1261] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260812 19:13:30.406881 21324 rpcz_store.cc:276] 0812 19:13:27.587292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:27.587916 (+ 624us) service_pool.cc:227] Handling call
0812 19:13:27.685375 (+ 97459us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:27.685825 (+ 450us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:27.685853 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:27.686379 (+ 526us) log.cc:842] Serialized 26 byte log entry
0812 19:13:27.883872 (+197493us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:27.883914 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:27.883989 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:30.003713 (+2119724us) spinlock_profiling.cc:243] Waited 7 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:30.331939 (+328226us) raft_consensus.cc:1671] finished
0812 19:13:30.331962 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:30.339088 (+ 7126us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1680,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":12438,"spinlock_wait_cycles":16091392,"thread_start_us":197081,"threads_started":1,"wal-append.queue_time_us":222483}
I20260812 19:13:30.480635 23676 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:30.465631 21322 raft_consensus.cc:2454] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:30.534267 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:30.548666 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 46376.4us
I20260812 19:13:30.548998 21238 raft_consensus.cc:2379] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:30.534987 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:30.549902 21318 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:30.532073 21327 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:30.537397 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:30.541323 22011 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:30.461357 23855 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:30.552098 21243 raft_consensus.cc:1226] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260812 19:13:30.548224 21237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1509) took 1018 ms. Trace:
I20260812 19:13:30.554534 22052 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:30.426602 23602 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:30.555385 21154 raft_consensus.cc:1261] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:30.554553 21237 rpcz_store.cc:276] 0812 19:13:29.529646 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.519787 (+990141us) service_pool.cc:227] Handling call
0812 19:13:30.520235 (+ 448us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:30.547586 (+ 27351us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:30.548149 (+ 563us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:30.553603 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:30.535080 23585 raft_consensus.cc:2726] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:30.590062 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:30.428786 23832 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:30.564812 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:30.593928 21327 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:30.568944 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 108832us
I20260812 19:13:30.557799 21200 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:30.590497 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 47912.2us
I20260812 19:13:30.569177 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:30.580041 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:30.564607 21154 raft_consensus.cc:1226] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:30.597568 21322 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:30.595439 21114 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:30.565186 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.579622 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:13:30.582957 21287 connection.cc:232] RPC call timeout handler was delayed by -8.234662506962195s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1655s for a response
W20260812 19:13:30.600885 21287 connection.cc:232] RPC call timeout handler was delayed by -8.192362506990321s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17020000000000002s for a response
W20260812 19:13:30.601292 21287 connection.cc:232] RPC call timeout handler was delayed by -8.0600625069346279s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1849s for a response
W20260812 19:13:30.601703 21287 connection.cc:232] RPC call timeout handler was delayed by -8.02136250701733s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1892s for a response
I20260812 19:13:30.599047 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:30.604225 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.604650 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.591341 21321 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:30.605257 21233 raft_consensus.cc:2379] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
W20260812 19:13:30.575229 22008 leader_election.cc:334] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.987s (SENT)
I20260812 19:13:30.606000 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:30.606652 22008 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:30.606897 21321 raft_consensus.cc:3037] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:30.606768 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:30.607851 22008 leader_election.cc:334] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.018s (SENT)
I20260812 19:13:30.608014 21200 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:30.608304 22008 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:30.611950 21238 raft_consensus.cc:2354] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 4.
I20260812 19:13:30.552460 23605 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.554569 23735 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:30.616858 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:30.632069 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:30.633222 21322 raft_consensus.cc:3037] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:30.624401 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:30.624325 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.624756 21200 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:30.618574 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.615623 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:30.635403 21237 raft_consensus.cc:2379] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 4: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
W20260812 19:13:30.615924 21287 connection.cc:232] RPC call timeout handler was delayed by -7.9925625069299713s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19240000000000002s for a response
W20260812 19:13:30.636405 21287 connection.cc:232] RPC call timeout handler was delayed by -7.9718625069363043s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1947s for a response
W20260812 19:13:30.636794 21287 connection.cc:232] RPC call timeout handler was delayed by -7.9358625069726259s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19870000000000002s for a response
W20260812 19:13:30.637039 21287 connection.cc:232] RPC call timeout handler was delayed by -7.9169625069480389s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2008s for a response
W20260812 19:13:30.637270 21287 connection.cc:232] RPC call timeout handler was delayed by -7.8575625070370734s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2074s for a response
W20260812 19:13:30.637504 21287 connection.cc:232] RPC call timeout handler was delayed by -7.7900625070324168s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2149s for a response
W20260812 19:13:30.637741 21287 connection.cc:232] RPC call timeout handler was delayed by -7.191562506952323s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28140000000000004s for a response
W20260812 19:13:30.637990 21287 connection.cc:232] RPC call timeout handler was delayed by -6.8747625070391223s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3166s for a response
W20260812 19:13:30.638228 21287 connection.cc:232] RPC call timeout handler was delayed by -6.7028625069651753s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33570000000000005s for a response
I20260812 19:13:30.567875 23592 raft_consensus.cc:2726] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:13:30.641585 21153 raft_consensus.cc:2379] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:30.579113 23866 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:30.652330 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.654038 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:30.654786 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.655717 21233 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:30.642971 21200 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:30.558498 23821 raft_consensus.cc:2781] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260812 19:13:30.638460 21287 connection.cc:232] RPC call timeout handler was delayed by -3.670663914992474s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260812 19:13:30.642350 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.533742 23868 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:30.553517 23855 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.591449 23762 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:30.674993 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:30.567808 23751 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:30.537305 23788 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:30.697795 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:30.651281 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:30.699287 21158 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:30.606289 23759 raft_consensus.cc:2726] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:13:30.662417 21238 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:30.616353 23633 raft_consensus.cc:2781] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:30.629792 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:30.613914 23929 raft_consensus.cc:2726] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:30.686058 21153 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:30.686389 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 62901.9us
I20260812 19:13:30.685266 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:30.687469 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:30.644342 23735 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:30.465359 23856 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:30.691417 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.692981 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.731640 21243 raft_consensus.cc:2396] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260812 19:13:30.695024 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:30.647207 23791 raft_consensus.cc:2781] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:30.733142 23791 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:30.733755 23791 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:30.736560 21327 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:30.629698 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:30.698529 21233 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:30.660497 23605 consensus_queue.cc:1046] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
I20260812 19:13:30.651561 23585 raft_consensus.cc:2781] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:30.534116 23833 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:30.652626 23775 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.513612 23907 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:30.660192 23821 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:30.687052 23612 consensus_meta.cc:287] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 2.046s user 0.017s sys 0.000s
I20260812 19:13:30.703411 21318 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:30.721483 21287 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:30.720548 21155 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:30.729256 21114 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:30.730048 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:30.686679 23866 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:30.739233 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:30.747335 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.334712 23619 rpcz_store.cc:276] 0812 19:13:03.559179 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:03.567028 (+ 7849us) service_pool.cc:227] Handling call
0812 19:13:03.760054 (+193026us) log.cc:842] Serialized 506 byte log entry
0812 19:13:04.505394 (+745340us) spinlock_profiling.cc:243] Waited 992 us on lock 0x7b54000d1c20. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9a37a4f 00007f52d9a20545 00007f52d9a25771 00007f52d9a256f9 00007f52d9a25689 00007f52d9a25651 00007f52d9a2481d 00007f52dfde2204 00007f52dfde2039 00007f52d6b8a6b6
0812 19:13:30.271412 (+25766018us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9915,"raft.queue_time_us":856,"raft.run_cpu_time_us":12157,"raft.run_wall_time_us":762594,"spinlock_wait_cycles":2765056,"thread_start_us":142317,"threads_started":1,"wal-append.queue_time_us":154151,"wal-append.run_cpu_time_us":20564,"wal-append.run_wall_time_us":1133283}
I20260812 19:13:30.751223 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:30.703732 23608 raft_consensus.cc:2781] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:30.731624 21237 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:30.758005 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:30.760653 23923 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:30.676433 23827 consensus_queue.cc:1046] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260812 19:13:30.684222 23727 consensus_queue.cc:1035] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.068s
I20260812 19:13:30.701949 23751 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:30.787210 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.726202 23633 raft_consensus.cc:686] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:30.788024 21321 raft_consensus.cc:2396] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260812 19:13:30.882458 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:30.883482 21321 raft_consensus.cc:2379] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:30.879905 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:30.884541 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:30.884577 21155 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:30.885376 21321 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:30.782982 21327 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:30.886713 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:13:30.822829 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 93673.8us
I20260812 19:13:30.887820 21157 raft_consensus.cc:2379] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:30.830791 22011 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:30.885699 21243 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:13:30.888942 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:30.829017 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 107629us
I20260812 19:13:30.880806 21233 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:30.895196 21287 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:30.841207 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:30.785531 23821 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:30.907243 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:30.908110 21238 raft_consensus.cc:2354] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6.
I20260812 19:13:30.899349 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:30.909811 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 79126.4us
I20260812 19:13:30.772019 23849 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.910244 21233 raft_consensus.cc:1226] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260812 19:13:30.794860 23750 consensus_meta.cc:287] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.607s user 0.012s sys 0.008s
I20260812 19:13:30.901705 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:30.821507 23608 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:30.781339 23585 raft_consensus.cc:686] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:30.919719 23658 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:30.959046 23658 raft_consensus.cc:2726] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:30.785389 23884 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.961426 21158 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:30.923102 21233 raft_consensus.cc:1261] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260812 19:13:30.924153 23911 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.961905 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:30.915436 23901 log.cc:924] Time spent T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
I20260812 19:13:30.926343 23751 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:30.927628 23750 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:30.930733 23257 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:30.790338 23612 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:30.933141 23866 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.934123 21243 raft_consensus.cc:1226] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:30.935415 22011 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
W20260812 19:13:30.945967 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 57936.3us
W20260812 19:13:30.988937 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 53630.6us
I20260812 19:13:30.731451 23902 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:30.882366 23855 raft_consensus.cc:2726] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:30.955090 21151 raft_consensus.cc:1261] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260812 19:13:31.073382 21238 raft_consensus.cc:1226] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:31.074656 21158 raft_consensus.cc:1226] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260812 19:13:30.957254 23924 log.cc:924] Time spent T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.082s user 0.001s sys 0.001s
I20260812 19:13:30.957436 23585 consensus_queue.cc:237] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:30.879941 23676 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:30.890914 23898 raft_consensus.cc:2726] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:30.968104 21233 raft_consensus.cc:1261] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 3 index: 6. (term mismatch)
I20260812 19:13:31.103276 21233 pending_rounds.cc:76] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa: Aborting all ops after (but not including) 5
I20260812 19:13:31.103837 21233 pending_rounds.cc:97] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260812 19:13:31.104246 21233 raft_consensus.cc:2866] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:30.924862 23788 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:31.072685 22050 consensus_meta.cc:287] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.618s user 0.008s sys 0.009s
I20260812 19:13:30.986876 21236 raft_consensus.cc:2354] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 3. Current term is 4.
I20260812 19:13:30.988854 21114 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
W20260812 19:13:31.107371 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 118611us
I20260812 19:13:31.108140 21114 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:31.109539 21114 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:31.109645 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:30.946108 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 156378us
I20260812 19:13:31.110507 21238 raft_consensus.cc:2396] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260812 19:13:30.996501 21322 raft_consensus.cc:2454] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:30.997925 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:30.999722 22052 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 4.
I20260812 19:13:31.067561 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1512) took 1281 ms. Trace:
W20260812 19:13:31.070742 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 427833us
I20260812 19:13:30.947979 23602 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:30.934552 23803 raft_consensus.cc:2781] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:30.949601 23608 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:31.075621 21244 raft_consensus.cc:1226] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260812 19:13:31.089514 21112 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:31.096253 23868 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:30.986413 23750 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:30.961378 23928 raft_consensus.cc:2781] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:31.098322 21327 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
W20260812 19:13:31.073478 22008 leader_election.cc:334] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.969s (SENT)
I20260812 19:13:31.077869 23727 raft_consensus.cc:2781] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:30.978405 23907 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:31.067982 23911 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:31.120194 21288 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:13:31.192157 22008 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:31.196955 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 76883.3us
I20260812 19:13:31.117816 21243 rpcz_store.cc:276] 0812 19:13:29.786432 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.905744 (+1119312us) service_pool.cc:227] Handling call
0812 19:13:30.933940 (+ 28196us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:31.066856 (+132916us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:31.067473 (+ 617us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:31.117920 21200 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:31.119203 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:31.201803 23722 raft_consensus.cc:3032] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:31.202268 23722 raft_consensus.cc:729] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:31.202517 23911 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:31.203347 23722 consensus_queue.cc:260] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:31.204659 23722 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:31.200876 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:31.134435 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:31.134405 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:31.207062 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:31.207820 22052 raft_consensus.cc:2379] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:31.207934 21327 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:31.000626 23257 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:31.217120 21322 raft_consensus.cc:2454] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:31.102135 21237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1511) took 1398 ms. Trace:
I20260812 19:13:31.133831 23855 raft_consensus.cc:2781] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:30.909662 23923 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:31.209697 23856 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:31.212040 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:31.311672 21320 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:31.235344 23900 log.cc:924] Time spent T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260812 19:13:31.122464 23849 raft_consensus.cc:2781] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:31.162557 23925 raft_consensus.cc:2781] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:31.318445 23925 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:31.319031 23925 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:31.207108 21318 raft_consensus.cc:2454] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:31.121078 23819 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:31.121166 23912 raft_consensus.cc:2726] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260812 19:13:31.230106 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 112285us
I20260812 19:13:31.339469 21200 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:31.201851 23936 log.cc:924] Time spent T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260812 19:13:31.124327 23803 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:31.348498 23803 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:31.220059 21237 rpcz_store.cc:276] 0812 19:13:29.704386 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.890601 (+1186215us) service_pool.cc:227] Handling call
0812 19:13:30.891592 (+ 991us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:30.891886 (+ 294us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:30.938639 (+ 46753us) log.cc:842] Serialized 14 byte log entry
0812 19:13:31.101715 (+163076us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:31.101773 (+ 58us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:31.101840 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:31.101909 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:31.102065 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4262,"prepare.queue_time_us":57242,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":45,"raft.queue_time_us":4405,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":9088,"thread_start_us":158329,"threads_started":1,"wal-append.queue_time_us":256642}
W20260812 19:13:31.120438 23713 consensus_meta.cc:287] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.919s user 0.017s sys 0.000s
I20260812 19:13:31.133502 23832 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:31.199697 23788 raft_consensus.cc:2781] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260812 19:13:31.349723 23578 consensus_meta.cc:287] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.884s user 0.006s sys 0.006s
I20260812 19:13:31.237970 23750 raft_consensus.cc:2781] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election won for term 5
W20260812 19:13:31.284104 23935 log.cc:924] Time spent T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
I20260812 19:13:31.115603 23930 consensus_queue.cc:1035] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.182s
I20260812 19:13:31.293258 22046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 613) took 1009 ms. Trace:
I20260812 19:13:31.293615 23911 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:31.080822 23939 consensus_queue.cc:1046] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.220s
I20260812 19:13:31.198978 23928 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:31.221572 23898 raft_consensus.cc:2781] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:31.382933 23911 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260812 19:13:31.392778 23806 consensus_meta.cc:287] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.251s user 0.007s sys 0.005s
W20260812 19:13:31.308317 23937 log.cc:924] Time spent T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
W20260812 19:13:31.136329 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 47083.1us
W20260812 19:13:31.317821 23948 log.cc:924] Time spent T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
W20260812 19:13:31.329290 23891 log.cc:924] Time spent T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 1.602s user 0.008s sys 0.004s
W20260812 19:13:31.375417 23852 consensus_meta.cc:287] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.892s user 0.007s sys 0.010s
I20260812 19:13:31.336663 22055 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40598 (request call id 614) took 1051 ms. Trace:
I20260812 19:13:31.445454 22055 rpcz_store.cc:276] 0812 19:13:30.285599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.286075 (+ 476us) service_pool.cc:227] Handling call
0812 19:13:31.336291 (+1050216us) spinlock_profiling.cc:243] Waited 95 ms on lock 0x7b540058e654. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:31.336562 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":218420480}
I20260812 19:13:31.112488 23676 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:31.343480 21200 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706
W20260812 19:13:31.403751 23371 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.049s sys 0.130s
I20260812 19:13:31.485998 21112 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:31.486596 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 143240us
I20260812 19:13:31.237550 23855 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:31.328298 23850 consensus_meta.cc:287] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 2.150s user 0.014s sys 0.004s
I20260812 19:13:31.238767 21322 raft_consensus.cc:1261] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:13:31.363556 21323 raft_consensus.cc:1226] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: [3.3-5.4]
I20260812 19:13:31.366720 21149 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1771) took 3420 ms. Trace:
I20260812 19:13:31.363111 23788 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:31.312543 23833 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:31.372095 23459 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:31.375380 23750 raft_consensus.cc:686] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:31.351646 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:31.386699 23928 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:31.387984 22048 raft_consensus.cc:1226] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: []
W20260812 19:13:31.389117 23791 consensus_meta.cc:287] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.655s user 0.008s sys 0.004s
I20260812 19:13:31.099273 23633 consensus_queue.cc:237] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:31.394846 21156 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1763) took 3618 ms (client timeout 1971 ms). Trace:
I20260812 19:13:31.332935 23770 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:31.443392 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:13:31.452438 22055 raft_consensus.cc:1261] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260812 19:13:31.353300 21153 consensus_meta.cc:287] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.633s user 0.012s sys 0.000s
I20260812 19:13:31.356535 23751 raft_consensus.cc:2781] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election won for term 5
W20260812 19:13:31.466964 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 388434us
I20260812 19:13:31.376605 23902 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:31.355449 23819 consensus_queue.cc:1046] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.041s
I20260812 19:13:31.393324 23806 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:31.365494 23713 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:32.079607 21327 consensus_meta.cc:287] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.871s user 0.020s sys 0.008s
I20260812 19:13:32.080137 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1773) took 1858 ms. Trace:
I20260812 19:13:31.373004 22046 rpcz_store.cc:276] 0812 19:13:30.283442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.285236 (+ 1794us) service_pool.cc:227] Handling call
0812 19:13:30.365135 (+ 79899us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:30.620407 (+255272us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:30.622920 (+ 2513us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:30.711033 (+ 88113us) log.cc:842] Serialized 26 byte log entry
0812 19:13:31.116955 (+405922us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:31.124005 (+ 7050us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:31.139247 (+ 15242us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:31.292823 (+153576us) raft_consensus.cc:1671] finished
0812 19:13:31.292862 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:31.293150 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":71420,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":172,"thread_start_us":420493,"threads_started":2,"wal-append.queue_time_us":426294}
I20260812 19:13:31.444248 23612 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:32.232229 22010 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.043s user 0.000s sys 0.013s
W20260812 19:13:31.682139 22056 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.856s user 0.011s sys 0.061s
I20260812 19:13:31.487646 21200 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:32.712438 21156 rpcz_store.cc:269] 0812 19:13:27.776702 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:27.778778 (+ 2076us) service_pool.cc:227] Handling call
0812 19:13:31.394477 (+3615699us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b54001a3454. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:31.394724 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3282141440}
W20260812 19:13:32.466528 21321 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.248s user 0.015s sys 0.012s
W20260812 19:13:32.714186 21321 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.496s user 0.019s sys 0.016s
I20260812 19:13:32.082427 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:32.715021 22056 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.889s user 0.023s sys 0.081s
I20260812 19:13:31.500593 21112 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:31.635877 21320 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
W20260812 19:13:32.716836 21320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1762) took 2539 ms (client timeout 3268 ms). Trace:
I20260812 19:13:32.080726 21322 rpcz_store.cc:276] 0812 19:13:30.221630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:31.229385 (+1007755us) service_pool.cc:227] Handling call
0812 19:13:31.238278 (+ 8893us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:32.079488 (+841210us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:32.079997 (+ 509us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:32.717490 21320 rpcz_store.cc:269] 0812 19:13:30.177158 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.210734 (+ 33576us) service_pool.cc:227] Handling call
0812 19:13:31.211818 (+1001084us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b5800271898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:13:32.716753 (+1504935us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6646528}
I20260812 19:13:32.080147 21327 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
W20260812 19:13:32.719008 21327 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1537) took 1649 ms (client timeout 1916 ms). Trace:
W20260812 19:13:32.719725 21327 rpcz_store.cc:269] 0812 19:13:31.069776 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:31.206674 (+136898us) service_pool.cc:227] Handling call
0812 19:13:32.718945 (+1512271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:32.720369 21153 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
W20260812 19:13:32.721371 21153 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1790) took 2065 ms (client timeout 1913 ms). Trace:
W20260812 19:13:32.721901 21153 rpcz_store.cc:269] 0812 19:13:30.656595 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.674665 (+ 18070us) service_pool.cc:227] Handling call
0812 19:13:32.721275 (+2046610us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:32.570230 22008 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.372s user 0.002s sys 0.015s
I20260812 19:13:32.680806 22053 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40614 (request call id 428) took 4457 ms. Trace:
W20260812 19:13:32.200496 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 75834.9us
W20260812 19:13:32.730499 22010 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.541s user 0.005s sys 0.013s
W20260812 19:13:32.237730 22011 leader_election.cc:334] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 1.526s (SENT)
W20260812 19:13:32.727065 22008 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.529s user 0.002s sys 0.015s
I20260812 19:13:32.732569 22055 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40614 (request call id 434) took 1634 ms. Trace:
I20260812 19:13:32.680684 22051 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 611) took 4483 ms. Trace:
I20260812 19:13:32.729012 22053 rpcz_store.cc:276] 0812 19:13:28.223882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:28.225017 (+ 1135us) service_pool.cc:227] Handling call
0812 19:13:32.679954 (+4454937us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:32.680177 (+ 223us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:32.680404 (+ 227us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:32.680432 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:32.680473 (+ 41us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:32.680514 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:32.680586 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:32.680736 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6560128}
I20260812 19:13:32.729218 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1521) took 2540 ms. Trace:
W20260812 19:13:32.736598 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 1.663e+06us
I20260812 19:13:32.733263 22055 rpcz_store.cc:276] 0812 19:13:31.097836 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:31.451417 (+353581us) service_pool.cc:227] Handling call
0812 19:13:31.451918 (+ 501us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:32.715690 (+1263772us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:32.732501 (+ 16811us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:32.739295 22008 connection.cc:232] RPC call timeout handler was delayed by -1.4441673320252448s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2194s for a response
W20260812 19:13:32.739935 22008 connection.cc:232] RPC call timeout handler was delayed by -1.0182255699764937s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
W20260812 19:13:32.536530 21115 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.425s user 0.007s sys 0.014s
W20260812 19:13:32.742497 21115 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.631s user 0.012s sys 0.014s
W20260812 19:13:32.742571 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 1.2553e+06us
W20260812 19:13:32.742909 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 1.63161e+06us
W20260812 19:13:32.078783 23651 consensus_meta.cc:287] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.729s user 0.020s sys 0.006s
W20260812 19:13:32.024690 23722 consensus_meta.cc:287] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.820s user 0.012s sys 0.002s
W20260812 19:13:32.095467 23925 consensus_meta.cc:287] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.776s user 0.009s sys 0.004s
I20260812 19:13:32.732594 22056 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 620) took 2122 ms. Trace:
I20260812 19:13:32.762341 22056 rpcz_store.cc:276] 0812 19:13:30.610402 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.726603 (+116201us) service_pool.cc:227] Handling call
0812 19:13:30.775274 (+ 48671us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:30.775524 (+ 250us) raft_consensus.cc:1520] Early marking committed up to index 6
0812 19:13:30.825909 (+ 50385us) log.cc:842] Serialized 14 byte log entry
0812 19:13:32.732191 (+1906282us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:32.732275 (+ 84us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:32.732349 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:32.732437 (+ 88us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:32.732546 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16642,"prepare.queue_time_us":52799,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"thread_start_us":1889194,"threads_started":1,"wal-append.queue_time_us":1695926}
W20260812 19:13:32.241017 21158 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.132s user 0.026s sys 0.035s
W20260812 19:13:32.410588 21114 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.300s user 0.012s sys 0.007s
W20260812 19:13:32.764647 21158 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.656s user 0.035s sys 0.041s
I20260812 19:13:32.737520 21323 rpcz_store.cc:276] 0812 19:13:30.188484 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.189074 (+ 590us) service_pool.cc:227] Handling call
0812 19:13:31.361128 (+1172054us) spinlock_profiling.cc:243] Waited 3.79 ms on lock 0x7b5800271898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:31.361319 (+ 191us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:32.728856 (+1367537us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:32.729136 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8715008}
I20260812 19:13:31.444998 23832 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:32.042706 23803 consensus_meta.cc:287] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.693s user 0.013s sys 0.000s
I20260812 19:13:32.750461 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1523) took 2095 ms. Trace:
I20260812 19:13:32.780362 21320 rpcz_store.cc:276] 0812 19:13:30.655521 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:32.742853 (+2087332us) service_pool.cc:227] Handling call
0812 19:13:32.749595 (+ 6742us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:32.749943 (+ 348us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:32.749985 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:32.750047 (+ 62us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:32.750113 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:32.750190 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:32.750366 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:32.751364 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 1.26373e+06us
W20260812 19:13:31.487762 23735 consensus_meta.cc:287] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.704s user 0.009s sys 0.003s
I20260812 19:13:32.733493 22051 rpcz_store.cc:276] 0812 19:13:28.197037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:28.225523 (+ 28486us) service_pool.cc:227] Handling call
0812 19:13:32.679559 (+4454036us) spinlock_profiling.cc:243] Waited 2.53 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:32.679863 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:32.680180 (+ 317us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:32.680218 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:32.680277 (+ 59us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:32.680338 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:32.680417 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:32.680594 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5821184}
W20260812 19:13:32.582015 22050 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.360s user 0.015s sys 0.027s
W20260812 19:13:32.633821 23727 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.331s user 0.003s sys 0.028s
W20260812 19:13:32.037750 23608 consensus_meta.cc:287] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.874s user 0.013s sys 0.000s
W20260812 19:13:32.731757 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 494041us
I20260812 19:13:32.745926 23899 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:32.088198 23855 raft_consensus.cc:3037] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:31.493299 23371 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.734s user 0.049s sys 0.130s
W20260812 19:13:32.764724 21114 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.655s user 0.013s sys 0.009s
W20260812 19:13:32.806592 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 1.69697e+06us
W20260812 19:13:32.230101 21287 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.318s user 0.010s sys 0.004s
W20260812 19:13:32.807862 21287 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.896s user 0.013s sys 0.005s
I20260812 19:13:32.752197 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:32.728420 23821 consensus_meta.cc:287] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 1.802s user 0.039s sys 0.051s
I20260812 19:13:32.083559 23788 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:32.772912 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1525) took 2087 ms. Trace:
W20260812 19:13:32.818568 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 1.92345e+06us
I20260812 19:13:32.819725 21323 rpcz_store.cc:276] 0812 19:13:30.685495 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:32.766024 (+2080529us) service_pool.cc:227] Handling call
0812 19:13:32.772295 (+ 6271us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:32.772521 (+ 226us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:32.772546 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:32.772585 (+ 39us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:32.772625 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:32.772696 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:32.772834 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:32.773640 23751 raft_consensus.cc:686] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:32.317292 23676 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:32.458977 23949 consensus_queue.cc:1035] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.349s
I20260812 19:13:31.523764 23827 raft_consensus.cc:2781] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:32.366008 23713 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:31.544749 23893 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:32.746300 23791 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:31.521947 23868 raft_consensus.cc:2781] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:32.778921 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:32.789626 21200 connection.cc:232] RPC call timeout handler was delayed by -1.1269940759520978s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1859s for a response
I20260812 19:13:32.746804 22045 raft_consensus.cc:3037] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 4
I20260812 19:13:32.094962 23658 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:32.215133 23898 raft_consensus.cc:686] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 4 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:32.795697 22051 raft_consensus.cc:1261] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260812 19:13:32.029722 23923 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:32.749831 23850 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:32.756600 23819 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:32.742152 23459 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:32.811975 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1526) took 2120 ms. Trace:
I20260812 19:13:32.919937 21320 rpcz_store.cc:276] 0812 19:13:30.692011 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:32.781418 (+2089407us) service_pool.cc:227] Handling call
0812 19:13:32.811197 (+ 29779us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:32.811491 (+ 294us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:32.811531 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:32.811592 (+ 61us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:32.811657 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:32.811729 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:32.811886 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:32.813794 22056 pending_rounds.cc:76] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: Aborting all ops after (but not including) 2
I20260812 19:13:32.808454 23608 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:31.448495 23852 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:32.782541 23920 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.428s user 0.008s sys 0.010s
I20260812 19:13:32.793275 23612 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:32.797863 22050 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.582s user 0.015s sys 0.032s
W20260812 19:13:32.805331 23727 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.563s user 0.007s sys 0.028s
W20260812 19:13:32.782691 23623 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.410s user 0.010s sys 0.010s
I20260812 19:13:31.637921 23257 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:32.839538 21287 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:32.775532 23750 consensus_queue.cc:237] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:32.868184 22053 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:32.872499 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:32.954689 21323 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 1.
I20260812 19:13:32.761122 23651 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:32.781697 23925 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:32.798427 23735 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:32.791893 23832 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:32.799952 23803 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:32.848541 23827 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:32.775976 23902 consensus_queue.cc:1035] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.101s
I20260812 19:13:32.759720 23770 raft_consensus.cc:2781] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:13:32.921813 22056 pending_rounds.cc:97] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260812 19:13:32.874116 23952 raft_consensus.cc:3032] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:32.850265 23957 raft_consensus.cc:2781] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:32.971431 22056 raft_consensus.cc:2866] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEARNER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:32.941810 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:32.817101 23899 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:32.890383 23868 raft_consensus.cc:686] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:32.890225 23923 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:32.918572 23713 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:32.933578 23920 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.576s user 0.010s sys 0.010s
I20260812 19:13:32.884608 23676 consensus_queue.cc:1035] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
W20260812 19:13:32.955444 23623 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.580s user 0.010s sys 0.011s
I20260812 19:13:32.920908 23819 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:32.883543 23893 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:32.882231 23658 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:32.955024 23950 log.cc:924] Time spent T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.374s user 0.000s sys 0.001s
I20260812 19:13:32.821945 23821 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:32.918937 23850 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:32.943372 23727 raft_consensus.cc:686] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:32.956439 23953 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:32.960371 23608 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:32.964776 23968 raft_consensus.cc:2726] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:32.957623 23958 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:32.976905 23827 raft_consensus.cc:3037] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:32.958709 23954 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:32.952215 23612 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:32.976284 23952 raft_consensus.cc:729] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:32.976513 23957 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:32.963850 23751 consensus_queue.cc:237] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:32.958845 23898 consensus_queue.cc:237] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:33.052591 23770 raft_consensus.cc:686] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:13:33.074075 23858 log.cc:924] Time spent T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 4.843s user 0.010s sys 0.041s
I20260812 19:13:33.011255 23923 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:33.076135 23962 log.cc:924] Time spent T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
I20260812 19:13:32.711289 21149 rpcz_store.cc:276] 0812 19:13:27.946327 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:27.946450 (+ 123us) service_pool.cc:227] Handling call
0812 19:13:27.946830 (+ 380us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:27.960910 (+ 14080us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:27.960942 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:27.993530 (+ 32588us) log.cc:842] Serialized 26 byte log entry
0812 19:13:28.286979 (+293449us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:28.287039 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:31.358485 (+3071446us) spinlock_profiling.cc:243] Waited 8.4 ms on lock 0x7b540030d9d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:31.358545 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:31.358604 (+ 59us) raft_consensus.cc:1671] finished
0812 19:13:31.358641 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:31.366629 (+ 7988us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":88680,"raft.queue_time_us":24082,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":19286,"spinlock_wait_cycles":19319552,"thread_start_us":107405,"threads_started":1,"wal-append.queue_time_us":252209,"wal-append.run_cpu_time_us":12885,"wal-append.run_wall_time_us":2279611}
I20260812 19:13:32.837247 21152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1751) took 5350 ms. Trace:
I20260812 19:13:32.874142 21200 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:32.872047 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:32.871131 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:32.869767 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:32.956284 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 116919us
W20260812 19:13:32.844553 21204 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.769s user 0.007s sys 0.062s
I20260812 19:13:32.886102 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:32.925423 21236 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57682 (request call id 218) took 2564 ms (client timeout 1526 ms). Trace:
I20260812 19:13:32.973315 21320 raft_consensus.cc:2454] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:32.965095 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:32.963311 22050 raft_consensus.cc:2932] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEARNER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:13:33.190126 23758 raft_consensus.cc:3032] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Stepping down as leader of term 4
W20260812 19:13:33.072352 23965 log.cc:924] Time spent T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.105s user 0.000s sys 0.001s
I20260812 19:13:33.599738 23758 raft_consensus.cc:729] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:33.600939 21287 leader_election.cc:398] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5.
I20260812 19:13:33.353175 23735 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:33.355907 23819 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:33.368281 23821 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:33.602895 21204 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.527s user 0.009s sys 0.074s
I20260812 19:13:33.365020 23893 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:33.052994 23713 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:33.168933 23425 raft_consensus.cc:3032] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:33.605192 23425 raft_consensus.cc:729] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:33.188745 23764 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:33.364283 23925 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:33.479933 21154 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:33.364286 23770 consensus_queue.cc:237] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:33.181244 23788 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:33.433149 23952 consensus_queue.cc:260] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:33.480988 21158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 256) took 2414 ms. Trace:
I20260812 19:13:33.177604 23806 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:33.366544 23850 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:33.081914 23897 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:32.920080 21203 leader_election.cc:334] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Timed out: RequestConsensusVote RPC to 127.19.70.129:36551 timed out after 1.670s (SENT)
I20260812 19:13:33.354451 23852 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:33.611297 21158 rpcz_store.cc:276] 0812 19:13:31.066984 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:31.087569 (+ 20585us) service_pool.cc:227] Handling call
0812 19:13:31.107702 (+ 20133us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:31.108088 (+ 386us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:31.108115 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:31.108724 (+ 609us) log.cc:842] Serialized 52 byte log entry
0812 19:13:32.765269 (+1656545us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:32.765329 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:32.765440 (+ 111us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:33.480207 (+714767us) raft_consensus.cc:1671] finished
0812 19:13:33.480249 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:33.480728 (+ 479us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7815,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":15429,"thread_start_us":1656066,"threads_started":1,"wal-append.queue_time_us":1773861}
I20260812 19:13:33.479060 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:13:33.943885 21236 rpcz_store.cc:269] 0812 19:13:30.360908 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.459425 (+ 98517us) service_pool.cc:227] Handling call
0812 19:13:32.925104 (+2465679us) spinlock_profiling.cc:243] Waited 453 ms on lock 0x7b54002f17d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:32.925325 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1041387264}
I20260812 19:13:34.400760 23923 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:33.478996 21150 raft_consensus.cc:2379] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:33.960096 21323 raft_consensus.cc:2379] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:33.959333 22048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 621) took 3047 ms. Trace:
W20260812 19:13:33.988734 21235 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1519) took 3834 ms (client timeout 1863 ms). Trace:
I20260812 19:13:33.513923 21324 raft_consensus.cc:2396] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260812 19:13:33.514335 21327 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.756s user 0.001s sys 0.013s
W20260812 19:13:33.524834 22011 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.709s user 0.027s sys 0.027s
I20260812 19:13:33.970638 21244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1505) took 4759 ms. Trace:
I20260812 19:13:34.272677 21243 raft_consensus.cc:2379] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 4: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:34.031709 21237 raft_consensus.cc:2396] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260812 19:13:34.023512 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1541) took 1139 ms. Trace:
W20260812 19:13:34.009054 21154 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51004 (request call id 252) took 3417 ms (client timeout 1988 ms). Trace:
W20260812 19:13:33.524590 22010 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.791s user 0.007s sys 0.008s
W20260812 19:13:33.524323 22008 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.783s user 0.000s sys 0.016s
W20260812 19:13:34.450557 21235 rpcz_store.cc:269] 0812 19:13:30.154149 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.286151 (+132002us) service_pool.cc:227] Handling call
0812 19:13:33.988100 (+3701949us) spinlock_profiling.cc:243] Waited 36.2 ms on lock 0x7b5400032354. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:33.988386 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":83202944}
I20260812 19:13:34.006537 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1504) took 4817 ms. Trace:
I20260812 19:13:34.011005 22056 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 627) took 1775 ms. Trace:
W20260812 19:13:33.511994 21322 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.004s sys 0.000s
W20260812 19:13:34.495380 22010 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.762s user 0.017s sys 0.022s
W20260812 19:13:34.002754 21239 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1495) took 6473 ms (client timeout 1903 ms). Trace:
I20260812 19:13:34.479475 21320 rpcz_store.cc:276] 0812 19:13:32.793505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:32.921478 (+127973us) service_pool.cc:227] Handling call
0812 19:13:33.913796 (+992318us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:33.946265 23875 log.cc:924] Time spent T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 4.878s user 0.014s sys 0.055s
W20260812 19:13:34.450233 21115 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.699s user 0.023s sys 0.000s
I20260812 19:13:33.962347 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:34.446220 21112 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.691s user 0.007s sys 0.016s
I20260812 19:13:33.534961 23658 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:34.373538 23762 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.742s user 0.005s sys 0.030s
W20260812 19:13:33.072402 23960 log.cc:924] Time spent T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.143s user 0.000s sys 0.000s
W20260812 19:13:33.373401 23928 consensus_meta.cc:287] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.620s user 0.006s sys 0.005s
I20260812 19:13:34.019481 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:34.633795 22045 consensus_meta.cc:287] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.744s user 0.007s sys 0.011s
I20260812 19:13:34.634245 22046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40614 (request call id 436) took 1033 ms. Trace:
I20260812 19:13:34.634913 22046 rpcz_store.cc:276] 0812 19:13:33.600632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:34.018689 (+418057us) service_pool.cc:227] Handling call
0812 19:13:34.634119 (+615430us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:34.631394 23371 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.759s user 0.004s sys 0.016s
W20260812 19:13:34.636896 23371 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.476s user 0.005s sys 0.017s
I20260812 19:13:34.637380 23757 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:34.489351 21240 rpcz_store.cc:276] 0812 19:13:29.189284 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:29.208125 (+ 18841us) service_pool.cc:227] Handling call
0812 19:13:29.991918 (+783793us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:30.172371 (+180453us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:34.005840 (+3833469us) spinlock_profiling.cc:243] Waited 63.2 ms on lock 0x7b5400150ed4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:34.006313 (+ 473us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":145469824}
I20260812 19:13:34.640311 23788 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:33.975744 23917 raft_consensus.cc:3032] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:34.641165 23917 raft_consensus.cc:729] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:34.634402 22045 raft_consensus.cc:2454] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
W20260812 19:13:34.549659 21235 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1508) took 1679 ms (client timeout 1641 ms). Trace:
W20260812 19:13:34.480471 21154 rpcz_store.cc:269] 0812 19:13:30.591457 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.615222 (+ 23765us) service_pool.cc:227] Handling call
0812 19:13:34.008708 (+3393486us) spinlock_profiling.cc:243] Waited 2.84 s on lock 0x7b54003cf354. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:34.008940 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6539409792}
W20260812 19:13:34.644284 21328 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1772) took 4422 ms (client timeout 2032 ms). Trace:
I20260812 19:13:33.551019 23961 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:34.644824 21328 rpcz_store.cc:269] 0812 19:13:30.221307 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.307133 (+ 85826us) service_pool.cc:227] Handling call
0812 19:13:34.635254 (+4328121us) spinlock_profiling.cc:243] Waited 336 ms on lock 0x7b540016dfd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:34.644058 (+ 8804us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":772607360}
I20260812 19:13:34.276597 23697 raft_consensus.cc:1013] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 3 LEADER]: attempt to promote peer 63db10063a7d4eb2a9cc9dc952654706: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260812 19:13:34.645974 23961 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:34.646873 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:34.643998 22045 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40598 (request call id 624) took 3643 ms (client timeout 2030 ms). Trace:
I20260812 19:13:34.632694 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
W20260812 19:13:34.647631 22045 rpcz_store.cc:269] 0812 19:13:31.001058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:31.234932 (+233874us) service_pool.cc:227] Handling call
0812 19:13:34.643850 (+3408918us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:34.648234 22051 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:34.491021 22056 rpcz_store.cc:276] 0812 19:13:32.235752 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:32.782401 (+546649us) service_pool.cc:227] Handling call
0812 19:13:32.813583 (+ 31182us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:32.972460 (+158877us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:32.972506 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:32.988089 (+ 15583us) log.cc:842] Serialized 26 byte log entry
0812 19:13:34.010102 (+1022013us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:34.010244 (+ 142us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:34.010371 (+ 127us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:34.010449 (+ 78us) raft_consensus.cc:1671] finished
0812 19:13:34.010465 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:34.010780 (+ 315us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":66180,"threads_started":1,"wal-append.queue_time_us":305028}
I20260812 19:13:34.643050 23917 consensus_queue.cc:260] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:34.652019 23917 raft_consensus.cc:3037] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:34.643350 22047 raft_consensus.cc:1226] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEARNER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:33.963590 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1498) took 6208 ms. Trace:
W20260812 19:13:34.502242 22011 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.687s user 0.046s sys 0.046s
W20260812 19:13:34.514680 22008 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.774s user 0.009s sys 0.031s
W20260812 19:13:34.449801 21323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1804) took 1517 ms (client timeout 1815 ms). Trace:
W20260812 19:13:34.402092 21233 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1507) took 1498 ms (client timeout 1511 ms). Trace:
I20260812 19:13:34.035037 21238 raft_consensus.cc:2454] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:13:34.475852 21244 rpcz_store.cc:276] 0812 19:13:29.211170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:29.219185 (+ 8015us) service_pool.cc:227] Handling call
0812 19:13:29.219723 (+ 538us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:31.121212 (+1901489us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:33.969706 (+2848494us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b54000b5754. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:33.970272 (+ 566us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4267345280}
I20260812 19:13:34.273399 23735 raft_consensus.cc:2781] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
W20260812 19:13:34.643887 21235 rpcz_store.cc:269] 0812 19:13:32.869846 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:34.549347 (+1679501us) service_pool.cc:214] Skipping call since client already timed out
0812 19:13:34.549546 (+ 199us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260812 19:13:34.505383 21322 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.552s user 0.010s sys 0.026s
W20260812 19:13:33.959164 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 1.88417e+06us
W20260812 19:13:33.950209 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 349922us
W20260812 19:13:34.672151 21115 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.921s user 0.023s sys 0.006s
I20260812 19:13:33.358358 23957 raft_consensus.cc:3037] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:34.274919 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 1.40063e+06us
W20260812 19:13:34.591135 21237 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1504) took 1722 ms (client timeout 1604 ms). Trace:
I20260812 19:13:34.529537 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1505) took 1660 ms. Trace:
I20260812 19:13:34.007921 21241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1508) took 4578 ms. Trace:
I20260812 19:13:34.401814 23459 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:34.656956 21233 rpcz_store.cc:269] 0812 19:13:32.869526 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:32.871763 (+ 2237us) service_pool.cc:227] Handling call
0812 19:13:34.322306 (+1450543us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:34.549376 21114 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.411s user 0.003s sys 0.014s
W20260812 19:13:34.497400 21239 rpcz_store.cc:269] 0812 19:13:27.529322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:27.529474 (+ 152us) service_pool.cc:227] Handling call
0812 19:13:34.002137 (+6472663us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400012554. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:34.002472 (+ 335us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2364029824}
W20260812 19:13:34.686175 21112 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.941s user 0.013s sys 0.018s
I20260812 19:13:34.459491 23713 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:34.656543 21323 rpcz_store.cc:269] 0812 19:13:32.932166 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:32.955496 (+ 23330us) service_pool.cc:227] Handling call
0812 19:13:34.449592 (+1494096us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:34.676141 21237 rpcz_store.cc:269] 0812 19:13:32.868520 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:32.870853 (+ 2333us) service_pool.cc:227] Handling call
0812 19:13:34.530077 (+1659224us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:33.386574 23855 consensus_meta.cc:287] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.564s user 0.010s sys 0.000s
W20260812 19:13:34.633802 21150 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1792) took 3946 ms (client timeout 1666 ms). Trace:
I20260812 19:13:33.369863 23803 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:34.549780 23879 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.010s user 0.000s sys 0.012s
W20260812 19:13:34.695420 21114 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.888s user 0.010s sys 0.016s
I20260812 19:13:34.459182 21152 rpcz_store.cc:276] 0812 19:13:27.486529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:27.492642 (+ 6113us) service_pool.cc:227] Handling call
0812 19:13:27.506233 (+ 13591us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:27.917307 (+411074us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:32.836805 (+4919498us) spinlock_profiling.cc:243] Waited 410 ms on lock 0x7b54001a3954. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:32.837129 (+ 324us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":943074176}
W20260812 19:13:33.168496 23963 log.cc:924] Time spent T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.177s user 0.000s sys 0.000s
I20260812 19:13:34.634920 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:34.696967 21235 raft_consensus.cc:1226] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260812 19:13:34.677435 21241 rpcz_store.cc:276] 0812 19:13:29.429898 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:29.430135 (+ 237us) service_pool.cc:227] Handling call
0812 19:13:29.993412 (+563277us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:30.411597 (+418185us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:34.007284 (+3595687us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b54000b6654. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:34.007691 (+ 407us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":785843072}
I20260812 19:13:34.549604 23952 raft_consensus.cc:3037] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:34.711107 21319 raft_consensus.cc:2362] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5. Re-sending same reply.
W20260812 19:13:34.712006 21319 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1763) took 4534 ms (client timeout 1859 ms). Trace:
W20260812 19:13:34.712517 21319 rpcz_store.cc:269] 0812 19:13:30.177599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.177772 (+ 173us) service_pool.cc:227] Handling call
0812 19:13:34.634401 (+4456629us) spinlock_profiling.cc:243] Waited 3.62 ms on lock 0x7b5800271898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:13:34.711910 (+ 77509us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8323840}
I20260812 19:13:34.714603 23899 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:34.697753 21324 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1774) took 4475 ms (client timeout 3034 ms). Trace:
W20260812 19:13:34.715308 21324 rpcz_store.cc:269] 0812 19:13:30.221986 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.558581 (+336595us) service_pool.cc:227] Handling call
0812 19:13:34.697385 (+4138804us) spinlock_profiling.cc:243] Waited 2.9 s on lock 0x7b54002e54d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:34.697639 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6680783744}
I20260812 19:13:33.359136 23953 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:34.716727 23849 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:34.710386 21235 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37894 (request call id 1510) took 1840 ms. Trace:
I20260812 19:13:34.717412 23849 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:34.717581 21235 rpcz_store.cc:276] 0812 19:13:32.870469 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:34.689837 (+1819368us) service_pool.cc:227] Handling call
0812 19:13:34.696414 (+ 6577us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:34.709986 (+ 13572us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:34.710288 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:34.449750 22048 rpcz_store.cc:276] 0812 19:13:30.911736 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.948396 (+ 36660us) service_pool.cc:227] Handling call
0812 19:13:31.132222 (+183826us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:32.742289 (+1610067us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:33.957230 (+1214941us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b5400611554. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:33.958588 (+ 1358us) spinlock_profiling.cc:243] Waited 46.5 ms on lock 0x7b5400611550. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:33.958978 (+ 390us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":660578432}
I20260812 19:13:34.676733 21243 rpcz_store.cc:276] 0812 19:13:32.868895 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:32.871654 (+ 2759us) service_pool.cc:227] Handling call
0812 19:13:34.501053 (+1629399us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:34.549081 23836 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.058s user 0.009s sys 0.006s
W20260812 19:13:34.692807 21238 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1506) took 1823 ms (client timeout 1760 ms). Trace:
W20260812 19:13:34.694892 23762 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.524s user 0.015s sys 0.041s
I20260812 19:13:34.696830 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:34.702772 21150 rpcz_store.cc:269] 0812 19:13:30.687740 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.713848 (+ 26108us) service_pool.cc:227] Handling call
0812 19:13:34.633338 (+3919490us) spinlock_profiling.cc:243] Waited 2.69 s on lock 0x7b540006ead4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:34.633675 (+ 337us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6191420672}
I20260812 19:13:34.684073 23697 raft_consensus.cc:1013] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 3 LEADER]: attempt to promote peer 63db10063a7d4eb2a9cc9dc952654706: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20260812 19:13:34.728574 21327 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.970s user 0.013s sys 0.025s
I20260812 19:13:33.319744 23651 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:34.730466 21238 rpcz_store.cc:269] 0812 19:13:32.869207 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:32.869393 (+ 186us) service_pool.cc:227] Handling call
0812 19:13:34.679316 (+1809923us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:34.719430 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:34.726534 23879 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.371s user 0.000s sys 0.021s
I20260812 19:13:33.357851 23791 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:34.733491 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 1.97462e+06us
I20260812 19:13:34.738763 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:33.639534 23758 consensus_queue.cc:260] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:34.718590 23953 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:34.737797 23836 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.382s user 0.011s sys 0.013s
I20260812 19:13:34.754189 22056 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 4.
I20260812 19:13:34.754276 21328 raft_consensus.cc:2454] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:34.751036 22055 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
W20260812 19:13:34.743183 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 1.9307e+06us
W20260812 19:13:34.763161 21328 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1800) took 1874 ms (client timeout 1943 ms). Trace:
W20260812 19:13:34.763656 21328 rpcz_store.cc:269] 0812 19:13:32.882171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:34.646379 (+1764208us) service_pool.cc:227] Handling call
0812 19:13:34.756872 (+110493us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:34.765189 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:33.429840 23954 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:34.766511 21328 raft_consensus.cc:2396] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
W20260812 19:13:34.767585 21328 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1787) took 1983 ms (client timeout 2568 ms). Trace:
W20260812 19:13:34.767861 23827 consensus_meta.cc:287] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.918s user 0.007s sys 0.007s
I20260812 19:13:34.768466 23827 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:34.771005 23827 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:34.771811 21288 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
W20260812 19:13:34.772390 23898 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.161s user 0.009s sys 0.017s
W20260812 19:13:34.772984 23898 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.337s user 0.009s sys 0.017s
I20260812 19:13:34.773049 23953 raft_consensus.cc:2781] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:34.773663 23953 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:34.774085 23953 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:34.774398 21318 raft_consensus.cc:1226] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 4 LEADER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:33.365643 23727 consensus_queue.cc:237] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:34.724763 23735 raft_consensus.cc:686] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:13:34.768164 21328 rpcz_store.cc:269] 0812 19:13:32.783857 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:34.764666 (+1980809us) service_pool.cc:227] Handling call
0812 19:13:34.767487 (+ 2821us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:34.780455 21320 raft_consensus.cc:3037] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:34.744773 23750 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.086s user 0.004s sys 0.008s
I20260812 19:13:33.430282 23958 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:34.791654 22055 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40598 (request call id 625) took 2545 ms (client timeout 2097 ms). Trace:
I20260812 19:13:34.654067 21329 rpcz_store.cc:276] 0812 19:13:27.757442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:27.791089 (+ 33647us) service_pool.cc:227] Handling call
0812 19:13:27.791608 (+ 519us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:27.841259 (+ 49651us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:27.841298 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:27.873953 (+ 32655us) log.cc:842] Serialized 52 byte log entry
0812 19:13:27.979036 (+105083us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:27.979077 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:27.979152 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:32.991846 (+5012694us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:33.962732 (+970886us) raft_consensus.cc:1671] finished
0812 19:13:33.962791 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:33.963262 (+ 471us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4864,"raft.queue_time_us":17722,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":13417,"spinlock_wait_cycles":45145472,"thread_start_us":81474,"threads_started":1,"wal-append.queue_time_us":170876}
W20260812 19:13:34.798997 22055 rpcz_store.cc:269] 0812 19:13:32.234912 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:32.750107 (+515195us) service_pool.cc:227] Handling call
0812 19:13:33.961632 (+1211525us) spinlock_profiling.cc:243] Waited 1.82 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:13:34.780076 (+818444us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4187392}
I20260812 19:13:34.012058 23425 consensus_queue.cc:260] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:34.813362 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:34.805532 23750 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.441s user 0.006s sys 0.008s
I20260812 19:13:34.842098 23827 consensus_queue.cc:1035] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260812 19:13:34.845908 21112 connection.cc:232] RPC call timeout handler was delayed by -2.0507408269913867s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1916s for a response
I20260812 19:13:34.842345 21114 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:34.848143 22008 leader_election.cc:334] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.526s (SENT)
I20260812 19:13:34.852705 21154 raft_consensus.cc:1226] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:34.854071 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:13:34.857201 22050 consensus_meta.cc:287] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.842s user 0.006s sys 0.009s
I20260812 19:13:34.820495 21287 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:34.862362 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1769) took 4645 ms. Trace:
I20260812 19:13:34.862677 21150 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:34.863726 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:34.865238 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:34.867708 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1775) took 4645 ms. Trace:
I20260812 19:13:34.869030 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:34.870541 22047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40614 (request call id 432) took 4585 ms. Trace:
I20260812 19:13:34.874744 22008 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:34.875700 22048 raft_consensus.cc:1226] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:34.875350 23735 consensus_queue.cc:237] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:34.881181 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:34.834233 23877 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.010s sys 0.036s
I20260812 19:13:34.818883 23986 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:34.877864 21318 rpcz_store.cc:276] 0812 19:13:30.222396 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:31.340977 (+1118581us) service_pool.cc:227] Handling call
0812 19:13:31.341417 (+ 440us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:34.780238 (+3438821us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:34.867248 (+ 87010us) spinlock_profiling.cc:243] Waited 3.43 s on lock 0x7b54003999d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:34.867584 (+ 336us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7894009088}
I20260812 19:13:33.291476 23868 consensus_queue.cc:237] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:34.883244 21244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1511) took 1989 ms (client timeout 1945 ms). Trace:
W20260812 19:13:34.943670 21244 rpcz_store.cc:269] 0812 19:13:32.872820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:34.694685 (+1821865us) service_pool.cc:227] Handling call
0812 19:13:34.861639 (+166954us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:34.912899 23877 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.934s user 0.011s sys 0.037s
I20260812 19:13:34.952827 21324 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:34.993268 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:34.993773 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:34.995146 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:34.995708 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:34.996146 23832 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:34.996739 21318 raft_consensus.cc:3037] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:34.877053 21321 rpcz_store.cc:276] 0812 19:13:30.214186 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.908886 (+694700us) service_pool.cc:227] Handling call
0812 19:13:30.909258 (+ 372us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:31.122992 (+213734us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:31.123039 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:31.123684 (+ 645us) log.cc:842] Serialized 26 byte log entry
0812 19:13:32.714768 (+1591084us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:32.714810 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:33.955935 (+1241125us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b54000d1e54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:33.956087 (+ 152us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:34.858657 (+902570us) raft_consensus.cc:1671] finished
0812 19:13:34.858688 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:34.859072 (+ 384us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10915,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":31210,"spinlock_wait_cycles":465712000,"thread_start_us":1496375,"threads_started":1,"wal-append.queue_time_us":1676174}
I20260812 19:13:34.996977 23612 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:34.970032 21324 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1795) took 2145 ms (client timeout 2152 ms). Trace:
W20260812 19:13:34.999204 21324 rpcz_store.cc:269] 0812 19:13:32.824572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:34.868566 (+2043994us) service_pool.cc:227] Handling call
0812 19:13:34.969944 (+101378us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:35.003984 21328 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:34.998418 23651 raft_consensus.cc:2726] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:35.029068 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:35.025372 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:35.033694 21321 raft_consensus.cc:1226] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:35.036520 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1527) took 4328 ms. Trace:
I20260812 19:13:35.037106 22056 raft_consensus.cc:1226] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:35.037019 21321 rpcz_store.cc:276] 0812 19:13:30.706477 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.017924 (+4311447us) service_pool.cc:227] Handling call
0812 19:13:35.033394 (+ 15470us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:35.034225 (+ 831us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.034445 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:34.998834 23961 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:34.999172 23697 raft_consensus.cc:2726] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:35.043502 22051 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
W20260812 19:13:35.049583 22051 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40614 (request call id 437) took 1448 ms (client timeout 1607 ms). Trace:
W20260812 19:13:35.034966 22011 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.70.129:36551 timed out after 1.988s (SENT) [suppressed 17 similar messages]
I20260812 19:13:35.037439 23425 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:35.050155 22051 rpcz_store.cc:269] 0812 19:13:33.601305 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:33.933203 (+331898us) service_pool.cc:227] Handling call
0812 19:13:35.049482 (+1116279us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:34.880084 22047 rpcz_store.cc:276] 0812 19:13:30.285462 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.300305 (+ 14843us) service_pool.cc:227] Handling call
0812 19:13:33.963269 (+3662964us) spinlock_profiling.cc:243] Waited 5.41 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:34.411811 (+448542us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:34.684997 (+273186us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:34.870250 (+185253us) spinlock_profiling.cc:243] Waited 47 ms on lock 0x7b54006e3050. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:34.870445 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":120536192}
I20260812 19:13:35.066424 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.037123 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:35.061419 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.066424 21324 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 5
I20260812 19:13:35.073217 21288 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:13:35.076949 21329 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
W20260812 19:13:35.085198 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 243017us
W20260812 19:13:35.094884 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 2.07649e+06us
I20260812 19:13:35.099292 21158 raft_consensus.cc:2379] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 4: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:13:35.099383 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 361674us
I20260812 19:13:35.076973 23758 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:35.099045 21319 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1796) took 2260 ms (client timeout 2097 ms). Trace:
W20260812 19:13:35.102586 21319 rpcz_store.cc:269] 0812 19:13:32.835597 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.075268 (+2239671us) service_pool.cc:214] Skipping call since client already timed out
0812 19:13:35.095492 (+ 20224us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260812 19:13:35.020138 23897 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:35.109911 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1777) took 4828 ms. Trace:
I20260812 19:13:35.110977 23850 raft_consensus.cc:2781] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election won for term 5
W20260812 19:13:35.118286 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 270205us
I20260812 19:13:35.047084 23855 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:35.106245 23852 raft_consensus.cc:2726] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:13:35.048383 23986 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:35.120522 21112 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:35.119490 23939 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:35.131454 21204 connection.cc:232] RPC call timeout handler was delayed by -2.4682526258984581s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1512s for a response
I20260812 19:13:35.130709 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1524) took 4455 ms. Trace:
I20260812 19:13:35.133180 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1528) took 4402 ms. Trace:
I20260812 19:13:35.133776 21321 rpcz_store.cc:276] 0812 19:13:30.724697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.059318 (+4334621us) service_pool.cc:227] Handling call
0812 19:13:35.110816 (+ 51498us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:35.121871 (+ 11055us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:35.121905 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:35.126838 (+ 4933us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:35.127011 (+ 173us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.127096 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:35.127217 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:35.075038 23928 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:35.148622 21320 raft_consensus.cc:2454] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
W20260812 19:13:35.147501 23764 consensus_meta.cc:287] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.457s user 0.007s sys 0.012s
I20260812 19:13:35.151235 21155 raft_consensus.cc:2396] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260812 19:13:35.152457 23918 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:35.153182 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:35.159113 21320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1797) took 2304 ms (client timeout 2030 ms). Trace:
W20260812 19:13:35.159709 21320 rpcz_store.cc:269] 0812 19:13:32.845005 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:34.718918 (+1873913us) service_pool.cc:227] Handling call
0812 19:13:35.149105 (+430187us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:35.161752 21204 connection.cc:232] RPC call timeout handler was delayed by -2.2579627640079707s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15280000000000002s for a response
I20260812 19:13:35.157582 21287 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:35.142759 23850 raft_consensus.cc:686] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:35.118961 21322 rpcz_store.cc:276] 0812 19:13:30.281559 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:32.743230 (+2461671us) service_pool.cc:227] Handling call
0812 19:13:32.774181 (+ 30951us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:32.774575 (+ 394us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:32.774602 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:32.775075 (+ 473us) log.cc:842] Serialized 26 byte log entry
0812 19:13:32.931009 (+155934us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:34.711918 (+1780909us) log.cc:842] Serialized 14 byte log entry
0812 19:13:35.084654 (+372736us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.097886 (+ 13232us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:35.107300 (+ 9414us) raft_consensus.cc:1671] finished
0812 19:13:35.107315 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:35.109851 (+ 2536us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8602,"prepare.queue_time_us":141750,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":38,"raft.queue_time_us":24739,"raft.run_cpu_time_us":462,"raft.run_wall_time_us":40419,"thread_start_us":2180932,"threads_started":3,"wal-append.queue_time_us":675694,"wal-append.run_cpu_time_us":2385,"wal-append.run_wall_time_us":1206574}
I20260812 19:13:35.180013 21151 raft_consensus.cc:2379] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 4: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:13:35.176056 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 95189.7us
I20260812 19:13:35.133244 21327 rpcz_store.cc:276] 0812 19:13:30.675841 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:32.743402 (+2067561us) service_pool.cc:227] Handling call
0812 19:13:32.743718 (+ 316us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:32.743955 (+ 237us) raft_consensus.cc:1520] Early marking committed up to index 6
0812 19:13:34.763472 (+2019517us) log.cc:842] Serialized 14 byte log entry
0812 19:13:35.104209 (+340737us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:35.108544 (+ 4335us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:35.119925 (+ 11381us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.123519 (+ 3594us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:35.130594 (+ 7075us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":852,"prepare.queue_time_us":328633,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":17439,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":11130,"thread_start_us":2292369,"threads_started":2,"wal-append.queue_time_us":362890}
I20260812 19:13:35.182881 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1529) took 4454 ms. Trace:
I20260812 19:13:35.183501 21329 rpcz_store.cc:276] 0812 19:13:30.728053 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.094896 (+4366843us) service_pool.cc:227] Handling call
0812 19:13:35.139776 (+ 44880us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:35.182408 (+ 42632us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:35.182436 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:35.182483 (+ 47us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:35.182518 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.182590 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:35.182818 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:35.123538 23849 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:35.195734 21236 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1509) took 2321 ms (client timeout 1810 ms). Trace:
W20260812 19:13:35.196369 21236 rpcz_store.cc:269] 0812 19:13:32.870129 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.175423 (+2305294us) service_pool.cc:214] Skipping call since client already timed out
0812 19:13:35.191594 (+ 16171us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260812 19:13:35.198009 23988 log.cc:924] Time spent T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20260812 19:13:35.200184 23720 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:35.201121 23720 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:35.205143 22008 connection.cc:232] RPC call timeout handler was delayed by -1.745696833007969s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2472s for a response
I20260812 19:13:35.142304 23939 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:35.205687 22008 connection.cc:232] RPC call timeout handler was delayed by -1.3560789888724685s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27530000000000004s for a response
W20260812 19:13:35.211716 22008 connection.cc:232] RPC call timeout handler was delayed by -1.0274449239950627s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3147s for a response
I20260812 19:13:35.208230 23850 consensus_queue.cc:237] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:35.163820 23658 consensus_queue.cc:1046] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.040s
I20260812 19:13:35.217900 21152 raft_consensus.cc:1226] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:35.215734 23985 consensus_queue.cc:1046] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.064s
I20260812 19:13:35.230517 23855 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:35.223189 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 103052us
I20260812 19:13:35.223894 23852 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:13:35.231666 23855 raft_consensus.cc:3032] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:35.153292 23918 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:35.232379 23852 raft_consensus.cc:686] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:35.232501 21287 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:35.213915 23697 raft_consensus.cc:2726] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:35.234086 23852 consensus_queue.cc:237] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:35.234920 23953 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:35.237504 23953 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:35.237802 21112 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:35.227625 23920 consensus_queue.cc:1046] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:13:35.228344 23958 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:35.228734 23954 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:35.239024 23997 log.cc:924] Time spent T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
I20260812 19:13:35.239881 23827 raft_consensus.cc:2781] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:35.240447 23827 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:35.240567 21318 raft_consensus.cc:2454] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:35.229146 23720 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:35.243237 21204 connection.cc:232] RPC call timeout handler was delayed by -1.9496982619166374s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20299999999999999s for a response
I20260812 19:13:35.240971 23827 raft_consensus.cc:3037] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:35.238266 23953 raft_consensus.cc:2726] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:35.271792 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:35.283150 23727 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:35.283455 22045 raft_consensus.cc:1226] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:35.284037 23727 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:35.286194 22053 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40614 (request call id 435) took 1604 ms. Trace:
I20260812 19:13:35.286906 23727 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:35.232329 23855 raft_consensus.cc:729] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:13:35.293951 21204 connection.cc:232] RPC call timeout handler was delayed by -1.7800638390472159s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2097s for a response
W20260812 19:13:35.294519 21204 connection.cc:232] RPC call timeout handler was delayed by -1.6842957239132375s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19410000000000002s for a response
I20260812 19:13:35.286700 22053 rpcz_store.cc:276] 0812 19:13:33.599844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:33.933057 (+333213us) service_pool.cc:227] Handling call
0812 19:13:34.033543 (+100486us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:34.460687 (+427144us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:34.460762 (+ 75us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:34.729651 (+268889us) log.cc:842] Serialized 52 byte log entry
0812 19:13:35.096812 (+367161us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:35.097086 (+ 274us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.097190 (+ 104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:35.203069 (+105879us) raft_consensus.cc:1671] finished
0812 19:13:35.203100 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:35.203491 (+ 391us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":416951,"raft.run_cpu_time_us":557,"raft.run_wall_time_us":175323,"thread_start_us":263406,"threads_started":1,"wal-append.queue_time_us":380713}
I20260812 19:13:35.300283 23720 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:35.308784 21203 leader_election.cc:334] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Timed out: RequestConsensusVote RPC to 127.19.70.129:36551 timed out after 1.735s (SENT)
I20260812 19:13:35.308724 23986 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:35.311271 21150 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:13:35.314785 21318 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 6
W20260812 19:13:35.319846 21323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1802) took 2397 ms (client timeout 2193 ms). Trace:
I20260812 19:13:35.286808 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
W20260812 19:13:35.320523 21323 rpcz_store.cc:269] 0812 19:13:32.922547 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.310767 (+2388220us) service_pool.cc:214] Skipping call since client already timed out
0812 19:13:35.319738 (+ 8971us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260812 19:13:35.322168 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 295533us
I20260812 19:13:35.310005 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:35.324317 22011 connection.cc:232] RPC call timeout handler was delayed by -1.0208739599911496s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1837s for a response
I20260812 19:13:35.309852 23720 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:35.326129 21324 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:35.328001 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:35.332355 22008 leader_election.cc:334] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.112s (SENT)
I20260812 19:13:35.330821 22045 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 6
I20260812 19:13:35.341326 23751 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:35.345687 22051 raft_consensus.cc:1226] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:35.339596 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:35.364524 21112 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:35.368690 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:35.365968 21323 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:35.369756 21244 raft_consensus.cc:2454] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:35.362840 23764 consensus_queue.cc:1046] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20260812 19:13:35.342226 23751 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:35.362437 23855 consensus_queue.cc:260] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:35.386951 23928 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:35.363718 23897 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:35.380095 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:35.350844 23952 consensus_meta.cc:287] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.639s user 0.011s sys 0.002s
I20260812 19:13:35.387986 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:35.375985 23757 consensus_meta.cc:287] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.738s user 0.015s sys 0.004s
I20260812 19:13:35.413502 23823 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:35.414232 23823 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:35.415827 23727 consensus_queue.cc:1046] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260812 19:13:35.417248 21157 raft_consensus.cc:1261] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:13:35.417279 23727 raft_consensus.cc:3032] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:35.418017 23727 raft_consensus.cc:729] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
W20260812 19:13:35.420536 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 111835us
I20260812 19:13:35.415978 22011 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:35.389060 23879 raft_consensus.cc:2781] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260812 19:13:35.434405 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 108852us
I20260812 19:13:35.435123 21238 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:35.430209 23879 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:35.439227 23879 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:35.439491 21324 raft_consensus.cc:1261] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260812 19:13:35.440197 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1790) took 2655 ms. Trace:
I20260812 19:13:35.440552 21324 rpcz_store.cc:276] 0812 19:13:32.784863 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.351219 (+2566356us) service_pool.cc:227] Handling call
0812 19:13:35.427192 (+ 75973us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:35.439892 (+ 12700us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.440127 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:35.442190 21150 raft_consensus.cc:1226] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 7. Ops: []
I20260812 19:13:35.442339 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.431032 23727 consensus_queue.cc:260] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:35.443295 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.399453 23849 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:35.448761 21153 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:35.449605 23720 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:35.450075 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:35.443990 21203 leader_election.cc:334] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Timed out: RequestConsensusVote RPC to 127.19.70.129:36551 timed out after 1.793s (SENT)
I20260812 19:13:35.430549 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:35.443028 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.449681 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:35.446360 23727 raft_consensus.cc:3037] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:35.416749 23658 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:35.429461 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 196474us
I20260812 19:13:35.463608 21239 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:35.478794 21241 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:35.452831 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1542) took 2648 ms. Trace:
I20260812 19:13:35.438254 23855 raft_consensus.cc:3037] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:35.485157 21149 raft_consensus.cc:2354] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 3.
W20260812 19:13:35.493914 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 168844us
I20260812 19:13:35.492177 21324 rpcz_store.cc:276] 0812 19:13:32.793846 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.441401 (+2647555us) service_pool.cc:227] Handling call
0812 19:13:35.441865 (+ 464us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:35.442248 (+ 383us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:35.442280 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:35.442344 (+ 64us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:35.442389 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.442451 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:35.442562 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:35.488149 21203 leader_election.cc:334] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Timed out: RequestConsensusVote RPC to 127.19.70.129:36551 timed out after 1.971s (SENT)
I20260812 19:13:35.489202 23877 consensus_queue.cc:1035] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:13:35.494685 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1789) took 2710 ms. Trace:
I20260812 19:13:35.498167 21329 rpcz_store.cc:276] 0812 19:13:32.784540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.388215 (+2603675us) service_pool.cc:227] Handling call
0812 19:13:35.388686 (+ 471us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:35.486272 (+ 97586us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:35.486298 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:35.494303 (+ 8005us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:35.494362 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.494435 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:35.494591 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:35.489908 23750 raft_consensus.cc:1050] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
W20260812 19:13:35.490151 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 74319.3us
I20260812 19:13:35.490851 23961 raft_consensus.cc:2781] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election won for term 6
W20260812 19:13:35.490932 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 91007us
I20260812 19:13:35.495878 21203 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:35.510422 21319 raft_consensus.cc:1226] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
W20260812 19:13:35.516316 22008 leader_election.cc:334] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.472s (SENT)
I20260812 19:13:35.517657 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.521667 22008 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:35.493517 23759 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:35.527565 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 295213us
I20260812 19:13:35.530422 21243 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:35.529465 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:35.533340 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.535267 21149 raft_consensus.cc:2379] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:35.536365 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.537843 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:35.509914 23750 raft_consensus.cc:1054] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Unable to promote non-voter 63db10063a7d4eb2a9cc9dc952654706: Illegal state: Leader has not yet committed an operation in its own term
I20260812 19:13:35.542737 23958 raft_consensus.cc:2726] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:35.547885 23751 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:35.548342 23920 consensus_queue.cc:1046] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260812 19:13:35.550045 23920 raft_consensus.cc:3032] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:35.485793 23764 consensus_queue.cc:1046] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.047s
I20260812 19:13:35.550606 21204 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:35.560154 22053 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:35.562543 22050 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 412) took 7630 ms. Trace:
I20260812 19:13:35.563304 22050 rpcz_store.cc:276] 0812 19:13:27.932476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:27.936702 (+ 4226us) service_pool.cc:227] Handling call
0812 19:13:30.380594 (+2443892us) spinlock_profiling.cc:243] Waited 50.8 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52bce 000000000034aea9 00007f52de972a30 00007f52dea387b4 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:30.380835 (+ 241us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:32.954883 (+2574048us) raft_consensus.cc:1520] Early marking committed up to index 2
0812 19:13:34.956606 (+2001723us) log.cc:842] Serialized 14 byte log entry
0812 19:13:35.373668 (+417062us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:35.434689 (+ 61021us) log.cc:842] Serialized 26 byte log entry
0812 19:13:35.434907 (+ 218us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:35.513312 (+ 78405us) log.cc:842] Serialized 14 byte log entry
0812 19:13:35.532643 (+ 19331us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.557720 (+ 25077us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:35.562180 (+ 4460us) raft_consensus.cc:1671] finished
0812 19:13:35.562209 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:35.562450 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4487,"raft.queue_time_us":222914,"raft.run_cpu_time_us":467,"raft.run_wall_time_us":8413,"spinlock_wait_cycles":116822912,"thread_start_us":2049481,"threads_started":3,"wal-append.queue_time_us":431813}
I20260812 19:13:35.567792 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:35.568089 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.568804 22047 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:35.572448 23993 log.cc:924] Time spent T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.411s user 0.001s sys 0.002s
I20260812 19:13:35.573463 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:35.573760 21149 raft_consensus.cc:2396] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260812 19:13:35.497346 23658 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:35.579289 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.581224 21156 raft_consensus.cc:2379] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:13:35.582077 23957 consensus_meta.cc:287] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.764s user 0.005s sys 0.010s
I20260812 19:13:35.582566 23957 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:35.584409 21240 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1512) took 2711 ms (client timeout 1826 ms). Trace:
I20260812 19:13:35.585001 23957 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:35.584869 21240 rpcz_store.cc:269] 0812 19:13:32.873159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:34.645717 (+1772558us) service_pool.cc:227] Handling call
0812 19:13:35.584127 (+938410us) spinlock_profiling.cc:243] Waited 850 ms on lock 0x7b54003bf454. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:35.584287 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1954659712}
I20260812 19:13:35.585897 22052 raft_consensus.cc:1261] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEARNER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260812 19:13:35.592818 23425 consensus_queue.cc:1046] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:13:35.595085 21287 leader_election.cc:398] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4.
I20260812 19:13:35.617456 21287 leader_election.cc:401] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:13:35.617871 23697 raft_consensus.cc:3032] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:35.618357 23697 raft_consensus.cc:729] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:35.620247 21287 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
W20260812 19:13:35.621999 23994 log.cc:924] Time spent T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.493s user 0.001s sys 0.002s
I20260812 19:13:35.622160 23681 raft_consensus.cc:2726] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4.
W20260812 19:13:35.595337 21155 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1794) took 4867 ms (client timeout 2568 ms). Trace:
I20260812 19:13:35.623116 23681 raft_consensus.cc:2726] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 19:13:35.623287 21155 rpcz_store.cc:269] 0812 19:13:30.727551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.901309 (+173758us) service_pool.cc:227] Handling call
0812 19:13:35.595018 (+4693709us) spinlock_profiling.cc:243] Waited 4.23 s on lock 0x7b5400146ed4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:35.595227 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9737485568}
W20260812 19:13:35.624966 23996 log.cc:924] Time spent T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.495s user 0.003s sys 0.000s
I20260812 19:13:35.627751 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1786) took 2844 ms. Trace:
I20260812 19:13:35.628643 21239 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:35.628332 21319 rpcz_store.cc:276] 0812 19:13:32.783491 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.188922 (+2405431us) service_pool.cc:227] Handling call
0812 19:13:35.216289 (+ 27367us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:35.510994 (+294705us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.627358 (+116364us) spinlock_profiling.cc:243] Waited 294 ms on lock 0x7b54002756d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:35.627620 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":675884416}
I20260812 19:13:35.590277 21150 raft_consensus.cc:1203] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[2.5-2.5] Dedup: 2.5->[]
I20260812 19:13:35.555023 23759 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:35.631428 21151 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51004 (request call id 258) took 2741 ms (client timeout 1889 ms). Trace:
W20260812 19:13:35.632091 21151 rpcz_store.cc:269] 0812 19:13:32.890521 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:33.477866 (+587345us) service_pool.cc:227] Handling call
0812 19:13:35.631160 (+2153294us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b54001a2a54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:35.631332 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1664089856}
I20260812 19:13:35.633658 23759 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:35.634574 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 56408.3us
W20260812 19:13:35.637516 21328 consensus_meta.cc:287] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.599s user 0.012s sys 0.001s
W20260812 19:13:35.596618 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 64885.4us
I20260812 19:13:35.597127 23751 raft_consensus.cc:2781] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:35.607070 23985 raft_consensus.cc:3032] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:35.582496 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:35.611135 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 100921us
I20260812 19:13:35.611650 21157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1779) took 5033 ms. Trace:
I20260812 19:13:35.651477 21156 raft_consensus.cc:2396] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260812 19:13:35.613059 23979 log.cc:924] Time spent T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.963s user 0.003s sys 0.012s
I20260812 19:13:35.614020 23823 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:35.615880 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:35.630968 21150 raft_consensus.cc:1226] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 4. Ops: [2.5-2.5]
W20260812 19:13:35.595849 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 107753us
I20260812 19:13:35.638711 21328 raft_consensus.cc:2396] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260812 19:13:35.546059 23879 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:35.551288 23961 raft_consensus.cc:686] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:35.635478 21112 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:35.637705 21287 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:35.652051 21238 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:35.653712 21318 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:35.649101 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1794) took 2839 ms. Trace:
I20260812 19:13:35.555145 23920 raft_consensus.cc:729] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:35.651091 21157 rpcz_store.cc:276] 0812 19:13:30.577835 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:30.936566 (+358731us) service_pool.cc:227] Handling call
0812 19:13:30.937005 (+ 439us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:35.417819 (+4480814us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.611263 (+193444us) spinlock_profiling.cc:243] Waited 4.48 s on lock 0x7b54003cf854. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:35.611517 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10301857024}
I20260812 19:13:35.660444 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.661434 21327 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:35.661993 23939 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:35.685570 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 48001us
I20260812 19:13:35.662566 23918 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:35.663056 22045 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:13:35.664297 23658 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:35.663345 21036 catalog_manager.cc:5495] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 1 to 2, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 2 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:35.650346 23985 raft_consensus.cc:729] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:35.667414 21154 raft_consensus.cc:2354] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 4.
I20260812 19:13:35.668663 21200 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:35.670830 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1801) took 2782 ms. Trace:
I20260812 19:13:35.699075 21287 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:35.699473 21150 raft_consensus.cc:1226] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:35.700207 23850 raft_consensus.cc:2726] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:35.702037 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:35.704532 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:35.581243 23764 raft_consensus.cc:3032] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:35.706102 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.706504 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.707361 21152 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 4.
I20260812 19:13:35.707496 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.708218 21155 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
W20260812 19:13:35.682189 22008 leader_election.cc:334] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.753s (SENT)
I20260812 19:13:35.708405 23697 consensus_queue.cc:260] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:35.709172 22008 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:35.682652 21321 raft_consensus.cc:1226] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:35.706125 23764 raft_consensus.cc:729] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:35.711943 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1531) took 4941 ms. Trace:
I20260812 19:13:35.711892 23764 consensus_queue.cc:260] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:35.712249 23920 consensus_queue.cc:260] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:35.712538 21321 rpcz_store.cc:276] 0812 19:13:30.770644 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.134742 (+4364098us) service_pool.cc:227] Handling call
0812 19:13:35.670734 (+535992us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:35.710519 (+ 39785us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.711668 (+ 1149us) spinlock_profiling.cc:243] Waited 490 ms on lock 0x7b54002305d0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:35.711844 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1128123264}
I20260812 19:13:35.680279 21204 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:35.709540 21155 raft_consensus.cc:1261] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260812 19:13:35.692874 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.715152 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:35.716609 22050 raft_consensus.cc:2379] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 4: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:35.716926 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1798) took 2862 ms. Trace:
I20260812 19:13:35.717334 21321 rpcz_store.cc:276] 0812 19:13:32.854717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.715860 (+2861143us) service_pool.cc:227] Handling call
0812 19:13:35.716284 (+ 424us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:35.716570 (+ 286us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:35.716596 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:35.716656 (+ 60us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:35.716700 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.716755 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:35.716864 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:35.718461 23986 raft_consensus.cc:3032] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:35.718899 23958 raft_consensus.cc:2726] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:35.719100 23986 raft_consensus.cc:729] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:35.683120 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.710111 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.720813 21157 raft_consensus.cc:2379] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 4: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:13:35.720991 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 85608.8us
I20260812 19:13:35.720469 23986 consensus_queue.cc:260] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:35.721639 23425 raft_consensus.cc:2781] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:35.721760 21112 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:35.721988 23986 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:35.722216 23425 raft_consensus.cc:686] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:35.723438 23425 consensus_queue.cc:237] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:35.676290 21319 rpcz_store.cc:276] 0812 19:13:32.810107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.629777 (+2819670us) service_pool.cc:227] Handling call
0812 19:13:35.630364 (+ 587us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:35.648608 (+ 18244us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:35.648639 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:35.648698 (+ 59us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:35.648752 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.648840 (+ 88us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:35.649021 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:35.725435 23939 raft_consensus.cc:2781] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:35.648351 23751 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:35.714301 22011 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:35.726765 21158 raft_consensus.cc:2354] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5.
I20260812 19:13:35.704164 23985 consensus_queue.cc:260] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:35.697957 21328 rpcz_store.cc:276] 0812 19:13:32.888939 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:34.785822 (+1896883us) service_pool.cc:227] Handling call
0812 19:13:35.670727 (+884905us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:35.708439 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.735158 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:35.734432 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 52294.8us
I20260812 19:13:35.737103 21319 raft_consensus.cc:1226] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260812 19:13:35.738670 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 48813us
W20260812 19:13:35.742388 21203 connection.cc:232] RPC call timeout handler was delayed by -2.1897877590963617s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19430000000000003s for a response
I20260812 19:13:35.686421 23899 raft_consensus.cc:2726] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:35.744727 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1799) took 2863 ms. Trace:
W20260812 19:13:35.756655 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 88192.7us
I20260812 19:13:35.757050 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1536) took 4688 ms. Trace:
I20260812 19:13:35.757555 21288 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:13:35.744668 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1793) took 2935 ms. Trace:
I20260812 19:13:35.757932 21320 rpcz_store.cc:276] 0812 19:13:31.068925 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.165516 (+4096591us) service_pool.cc:227] Handling call
0812 19:13:35.204036 (+ 38520us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:35.280984 (+ 76948us) raft_consensus.cc:1520] Early marking committed up to index 4
0812 19:13:35.427836 (+146852us) log.cc:842] Serialized 14 byte log entry
0812 19:13:35.756709 (+328873us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:35.756776 (+ 67us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:35.756818 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.756885 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:35.756991 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":231545,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"thread_start_us":307882,"threads_started":2}
I20260812 19:13:35.758163 21329 rpcz_store.cc:276] 0812 19:13:32.803641 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.503449 (+2699808us) service_pool.cc:227] Handling call
0812 19:13:35.533317 (+ 29868us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:35.689993 (+156676us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:35.690044 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:35.726313 (+ 36269us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:35.726360 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.738598 (+ 12238us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54002319d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:35.738669 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:35.738820 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":287951104}
I20260812 19:13:35.760125 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.758111 21319 rpcz_store.cc:276] 0812 19:13:32.876803 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.725247 (+2848444us) service_pool.cc:227] Handling call
0812 19:13:35.725800 (+ 553us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:35.739681 (+ 13881us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.739918 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:35.760958 21150 raft_consensus.cc:2354] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6.
W20260812 19:13:35.763079 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 106901us
I20260812 19:13:35.765589 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.765847 21036 catalog_manager.cc:5495] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:35.767386 24006 raft_consensus.cc:2781] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:13:35.767995 24006 raft_consensus.cc:686] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 6 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:35.763208 23958 raft_consensus.cc:2781] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:35.768612 23958 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:35.769021 23958 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:35.769129 24006 consensus_queue.cc:237] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:35.770915 23750 raft_consensus.cc:2781] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:35.739771 23751 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:35.754452 23868 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.004s sys 0.004s
I20260812 19:13:35.772712 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:35.744210 23697 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:35.770968 23720 raft_consensus.cc:2781] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:35.787989 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.767936 21036 catalog_manager.cc:5495] T dfa25b5175f14002a8b5ec8869d7c1d7 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 3 to 4, leader changed from 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 4 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:35.789069 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:35.793643 23720 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:35.794234 23720 raft_consensus.cc:3037] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:35.794950 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:35.788388 21243 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:35.793146 21327 raft_consensus.cc:2454] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:35.863145 23770 raft_consensus.cc:2726] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:35.863322 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:35.784467 23868 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.657s user 0.005s sys 0.004s
W20260812 19:13:35.864928 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 73335.4us
I20260812 19:13:35.827685 21320 raft_consensus.cc:3037] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 3
I20260812 19:13:35.866700 21318 raft_consensus.cc:1226] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:35.761890 23939 raft_consensus.cc:686] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:13:35.867554 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 82068.9us
W20260812 19:13:35.896687 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 111433us
I20260812 19:13:35.867980 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:35.794242 23920 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:35.762060 21238 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "f741102bfe1441d1b453e5a838649059"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:13:35.873982 21036 catalog_manager.cc:5495] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 2 to 3, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
W20260812 19:13:35.879902 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 87713.1us
W20260812 19:13:35.907491 21203 connection.cc:232] RPC call timeout handler was delayed by -2.05015658098273s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19410000000000002s for a response
I20260812 19:13:35.786593 23764 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:35.871102 23954 consensus_queue.cc:1046] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260812 19:13:35.870210 21154 raft_consensus.cc:2454] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:35.894088 21328 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:35.899856 23985 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:35.900188 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:35.893554 21319 raft_consensus.cc:1226] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:35.993860 21238 raft_consensus.cc:595] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Received request to transfer leadership to TS f741102bfe1441d1b453e5a838649059
I20260812 19:13:35.994308 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1544) took 3171 ms. Trace:
I20260812 19:13:35.913395 23961 consensus_queue.cc:237] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:35.997057 21238 raft_consensus.cc:603] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:13:35.997424 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1547) took 1151 ms. Trace:
W20260812 19:13:35.994699 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 95207.9us
I20260812 19:13:35.998494 21319 rpcz_store.cc:276] 0812 19:13:34.845789 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.761045 (+915256us) service_pool.cc:227] Handling call
0812 19:13:35.784856 (+ 23811us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:35.997023 (+212167us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.997341 (+ 318us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:35.997087 21318 rpcz_store.cc:276] 0812 19:13:32.822584 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.678853 (+2856269us) service_pool.cc:227] Handling call
0812 19:13:35.734458 (+ 55605us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:35.927252 (+192794us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.994243 (+ 66991us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:35.895354 23827 consensus_meta.cc:287] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.620s user 0.014s sys 0.009s
I20260812 19:13:35.894730 23750 raft_consensus.cc:686] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:36.029013 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:36.040827 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:36.024238 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:36.044416 21328 raft_consensus.cc:2396] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260812 19:13:36.021445 21200 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:36.045341 22051 raft_consensus.cc:2454] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:36.047204 21158 raft_consensus.cc:2379] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:36.040160 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1803) took 3110 ms. Trace:
I20260812 19:13:36.047416 21150 raft_consensus.cc:2354] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:13:36.048317 21319 raft_consensus.cc:1226] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:36.047787 21321 rpcz_store.cc:276] 0812 19:13:32.929970 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.734312 (+2804342us) service_pool.cc:227] Handling call
0812 19:13:35.866161 (+131849us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:36.039813 (+173652us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:36.039839 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:36.039885 (+ 46us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:36.039919 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:36.039980 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:36.040088 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:36.053859 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1548) took 1203 ms. Trace:
I20260812 19:13:36.054225 21319 rpcz_store.cc:276] 0812 19:13:34.846169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:36.013802 (+1167633us) service_pool.cc:227] Handling call
0812 19:13:36.029087 (+ 15285us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:36.048868 (+ 19781us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:36.049056 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:36.013477 22045 raft_consensus.cc:1226] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEARNER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:36.024047 21036 catalog_manager.cc:5495] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 2 to 3, leader changed from 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:13:36.057484 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:36.060389 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:36.056821 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1791) took 3271 ms. Trace:
I20260812 19:13:36.056532 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:36.057291 21204 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:36.058171 22011 leader_election.cc:398] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:13:36.062068 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:36.061543 22011 leader_election.cc:401] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:13:36.063607 21149 raft_consensus.cc:2379] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 4: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:36.073336 23877 raft_consensus.cc:2781] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:36.074105 23877 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:36.074259 24027 log.cc:924] Time spent T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260812 19:13:36.074662 23877 raft_consensus.cc:3037] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:36.053221 23879 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:36.081521 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:36.082571 21319 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:36.039933 23720 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:36.087086 23720 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:36.088131 21203 connection.cc:232] RPC call timeout handler was delayed by -1.7203992530703545s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21930000000000002s for a response
W20260812 19:13:36.088622 21203 connection.cc:232] RPC call timeout handler was delayed by -1.4278811400290579s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1815s for a response
I20260812 19:13:36.050076 23827 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:36.050504 22047 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
W20260812 19:13:36.066540 24025 log.cc:924] Time spent T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20260812 19:13:36.114878 21150 raft_consensus.cc:3032] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:36.083343 21036 catalog_manager.cc:5495] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 3 to 4, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 4 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:36.124539 21150 raft_consensus.cc:729] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:36.123847 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:36.058275 23612 consensus_queue.cc:1046] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.069s
W20260812 19:13:36.127079 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 69373.8us
I20260812 19:13:36.188930 21329 raft_consensus.cc:1226] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:36.125651 23827 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:36.221875 21036 catalog_manager.cc:5495] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 1 to 3, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
W20260812 19:13:36.242046 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 90998.9us
I20260812 19:13:36.149179 23727 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:36.234951 21150 consensus_queue.cc:260] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:36.246261 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 149761us
I20260812 19:13:36.305110 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:36.188279 23954 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 5
I20260812 19:13:36.329187 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:36.330729 21157 raft_consensus.cc:2379] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:36.332278 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1810) took 1047 ms. Trace:
I20260812 19:13:36.333338 21328 raft_consensus.cc:1261] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260812 19:13:36.333024 21327 rpcz_store.cc:276] 0812 19:13:35.285085 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:36.020952 (+735867us) service_pool.cc:227] Handling call
0812 19:13:36.055971 (+ 35019us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:36.322850 (+266879us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:36.322900 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:36.322967 (+ 67us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:36.331881 (+ 8914us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:36.331999 (+ 118us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:36.332170 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":80450}
I20260812 19:13:36.338184 21320 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:36.338402 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:36.339257 21156 raft_consensus.cc:2379] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:36.339643 21150 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:36.344141 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 294731us
I20260812 19:13:36.357487 21327 raft_consensus.cc:1226] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5. Ops: []
W20260812 19:13:36.357795 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 267466us
I20260812 19:13:36.353057 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:36.363696 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:36.343312 23697 consensus_meta.cc:287] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.543s user 0.009s sys 0.011s
I20260812 19:13:36.370342 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:36.371204 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:36.129947 23939 consensus_queue.cc:237] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:36.357926 21036 catalog_manager.cc:5495] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 2 to 4, leader changed from 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 4 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:13:36.376417 21329 raft_consensus.cc:2454] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
I20260812 19:13:36.375326 22011 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:36.382593 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:36.385180 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:36.336858 23727 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:36.389459 21156 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:36.389546 21322 raft_consensus.cc:1261] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260812 19:13:36.392957 21036 catalog_manager.cc:5495] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 1 to 3, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260812 19:13:36.386005 23986 consensus_meta.cc:287] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.646s user 0.009s sys 0.006s
I20260812 19:13:36.395459 21322 pending_rounds.cc:76] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412: Aborting all ops after (but not including) 2
I20260812 19:13:36.401505 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:36.382267 22050 raft_consensus.cc:1261] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260812 19:13:36.400061 21112 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:36.403779 21149 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:36.385031 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:36.409314 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:36.390861 23762 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:36.412712 21157 raft_consensus.cc:2379] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 4: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:36.412609 21150 raft_consensus.cc:1261] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260812 19:13:36.408926 21322 pending_rounds.cc:97] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260812 19:13:36.393568 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:36.416505 21322 raft_consensus.cc:2866] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:36.420209 23432 consensus_queue.cc:1046] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.087s
I20260812 19:13:36.408322 21036 catalog_manager.cc:5495] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 2 to 3, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:13:36.420379 24035 consensus_queue.cc:1046] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:36.350390 23750 consensus_queue.cc:237] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:36.416599 23958 consensus_meta.cc:287] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.647s user 0.013s sys 0.000s
I20260812 19:13:36.436836 23727 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:36.417279 23735 raft_consensus.cc:2781] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:36.428584 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:36.443922 21152 raft_consensus.cc:1261] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260812 19:13:36.440356 23751 consensus_meta.cc:287] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.574s user 0.005s sys 0.008s
I20260812 19:13:36.450572 21157 raft_consensus.cc:2396] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260812 19:13:36.401134 23855 consensus_queue.cc:1035] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.031s
I20260812 19:13:36.427663 23986 raft_consensus.cc:2781] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:36.456715 21036 catalog_manager.cc:5495] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 2 to 3, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:13:36.450094 23735 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:36.441099 24035 consensus_queue.cc:1225] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer f741102bfe1441d1b453e5a838649059 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260812 19:13:36.489028 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:36.061254 21323 rpcz_store.cc:276] 0812 19:13:32.785201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.381972 (+2596771us) service_pool.cc:227] Handling call
0812 19:13:35.451810 (+ 69838us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:35.452137 (+ 327us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:35.530145 (+ 78008us) log.cc:842] Serialized 14 byte log entry
0812 19:13:35.620320 (+ 90175us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b180035a040. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a638 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
0812 19:13:36.056431 (+436111us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:36.056488 (+ 57us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:36.056539 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:36.056610 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:36.056754 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7459,"prepare.queue_time_us":86803,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":56,"raft.queue_time_us":14687,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":8713,"spinlock_wait_cycles":27407232,"thread_start_us":570277,"threads_started":2}
I20260812 19:13:36.490048 21150 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:36.464684 22045 raft_consensus.cc:2379] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 4: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:36.495806 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:36.491947 21036 catalog_manager.cc:5495] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 1 to 3, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:36.454006 23697 consensus_queue.cc:1035] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260812 19:13:36.469534 23735 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:36.469249 23986 raft_consensus.cc:686] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:13:36.508152 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 116373us
I20260812 19:13:36.509863 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:36.453682 23751 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:36.453503 23958 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:36.503397 23920 consensus_meta.cc:287] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.598s user 0.010s sys 0.004s
I20260812 19:13:36.512526 21036 catalog_manager.cc:5495] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 2 to 4, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 4 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:36.532174 21153 raft_consensus.cc:2396] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260812 19:13:36.530057 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 288539us
I20260812 19:13:36.539422 22011 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:36.541611 21320 raft_consensus.cc:1261] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:36.546895 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:36.547729 24031 log.cc:924] Time spent T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.185s user 0.000s sys 0.001s
I20260812 19:13:36.551192 23751 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:36.551903 23751 raft_consensus.cc:3032] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:36.552332 23751 raft_consensus.cc:729] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:36.550376 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:36.560235 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:36.561971 21319 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
W20260812 19:13:36.540416 24028 log.cc:924] Time spent T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.201s user 0.000s sys 0.002s
I20260812 19:13:36.563803 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:36.566713 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:36.568315 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:36.569065 22056 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:36.569008 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:36.587500 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:36.577596 21036 catalog_manager.cc:5495] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:36.593752 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:36.566922 23764 consensus_meta.cc:287] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.658s user 0.014s sys 0.000s
W20260812 19:13:36.597594 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 63746.8us
I20260812 19:13:36.594223 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:36.566891 23432 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:36.604199 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:36.607177 21243 raft_consensus.cc:1261] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260812 19:13:36.570703 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:36.625128 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:36.636029 22046 raft_consensus.cc:1261] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260812 19:13:36.637214 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1543) took 3843 ms. Trace:
I20260812 19:13:36.639328 23759 consensus_queue.cc:1035] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:13:36.639942 24038 consensus_queue.cc:1035] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:13:36.637702 21324 rpcz_store.cc:276] 0812 19:13:32.794205 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.496082 (+2701877us) service_pool.cc:227] Handling call
0812 19:13:35.553294 (+ 57212us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:35.682586 (+129292us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:35.682649 (+ 63us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:35.689655 (+ 7006us) log.cc:842] Serialized 26 byte log entry
0812 19:13:35.894389 (+204734us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:35.894434 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:35.894507 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:36.636931 (+742424us) raft_consensus.cc:1671] finished
0812 19:13:36.636959 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:36.637119 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7784,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":28701,"thread_start_us":193726,"threads_started":1,"wal-append.queue_time_us":425141}
I20260812 19:13:36.640659 24038 consensus_queue.cc:1225] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer f741102bfe1441d1b453e5a838649059 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260812 19:13:36.612641 21036 catalog_manager.cc:5495] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:36.611285 23961 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election won for term 5
W20260812 19:13:36.651081 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 283124us
I20260812 19:13:36.652092 21200 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:36.653100 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:36.655251 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:36.657634 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:36.658546 21154 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
W20260812 19:13:36.664654 24024 log.cc:924] Time spent T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.638s user 0.003s sys 0.000s
I20260812 19:13:36.655658 21200 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:36.566478 23751 consensus_queue.cc:260] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:36.669266 22051 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40624 (request call id 439) took 1112 ms. Trace:
I20260812 19:13:36.675257 22051 rpcz_store.cc:276] 0812 19:13:35.556947 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.557130 (+ 183us) service_pool.cc:227] Handling call
0812 19:13:36.654549 (+1097419us) spinlock_profiling.cc:243] Waited 327 ms on lock 0x7b54005974d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:36.669192 (+ 14643us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":751992064}
I20260812 19:13:36.662218 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1788) took 3878 ms. Trace:
I20260812 19:13:36.680831 21322 rpcz_store.cc:276] 0812 19:13:32.784202 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.686161 (+2901959us) service_pool.cc:227] Handling call
0812 19:13:35.781956 (+ 95795us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:36.439254 (+657298us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:36.661831 (+222577us) spinlock_profiling.cc:243] Waited 601 ms on lock 0x7b540039add4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:36.662116 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1382054272}
I20260812 19:13:36.699174 21156 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
W20260812 19:13:36.699855 23877 consensus_meta.cc:287] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.625s user 0.002s sys 0.012s
W20260812 19:13:36.695333 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 107993us
I20260812 19:13:36.671573 21036 catalog_manager.cc:5495] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 2 to 3, leader changed from 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:36.700421 23877 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:36.712368 21243 raft_consensus.cc:1226] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: [1.2-3.3]
I20260812 19:13:36.671089 22045 raft_consensus.cc:2379] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 6: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:36.735020 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:36.705030 24029 log.cc:924] Time spent T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.263s user 0.002s sys 0.000s
I20260812 19:13:36.612459 23986 consensus_queue.cc:237] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:36.624419 23432 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:36.626430 24032 consensus_queue.cc:1046] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.099s
W20260812 19:13:36.744921 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 116771us
I20260812 19:13:36.695730 23961 raft_consensus.cc:686] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:36.727492 24040 raft_consensus.cc:2781] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:36.749600 21237 raft_consensus.cc:1226] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:36.668506 24011 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.070s
I20260812 19:13:36.742161 23918 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:36.757117 21235 raft_consensus.cc:1261] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260812 19:13:36.738102 23958 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:36.763006 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 96385us
I20260812 19:13:36.763707 21156 raft_consensus.cc:1226] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:36.799149 21154 raft_consensus.cc:1261] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260812 19:13:36.799835 23432 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:36.803296 21200 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:36.752017 24040 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:36.763007 23868 raft_consensus.cc:2781] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:36.799190 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:36.807720 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:36.819327 21155 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:36.754021 24052 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:36.822201 21156 pending_rounds.cc:76] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059: Aborting all ops after (but not including) 4
I20260812 19:13:36.767693 21036 catalog_manager.cc:5495] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 1 to 2, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 2 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } }
I20260812 19:13:36.851052 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:36.851663 21156 pending_rounds.cc:97] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20260812 19:13:36.762812 24032 consensus_queue.cc:1225] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer 93578836968649f6bfcaecc709871412 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260812 19:13:36.852229 21156 raft_consensus.cc:2866] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:36.853281 21235 raft_consensus.cc:2379] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:36.817515 24040 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:36.817421 23868 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:36.881525 21237 raft_consensus.cc:2379] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:36.822157 23751 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:13:36.853615 24042 log.cc:924] Time spent T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.287s user 0.000s sys 0.002s
I20260812 19:13:36.910794 22052 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40614 (request call id 442) took 1202 ms. Trace:
I20260812 19:13:36.785225 23918 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:36.926116 21036 catalog_manager.cc:5495] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 2 to 3, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:36.932552 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:36.927160 22052 rpcz_store.cc:276] 0812 19:13:35.708839 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.738390 (+ 29551us) service_pool.cc:227] Handling call
0812 19:13:35.763507 (+ 25117us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:35.763766 (+ 259us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:13:35.763798 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:35.797983 (+ 34185us) log.cc:842] Serialized 26 byte log entry
0812 19:13:36.047454 (+249471us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:36.047497 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:36.047560 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:36.910515 (+862955us) raft_consensus.cc:1671] finished
0812 19:13:36.910543 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:36.910723 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":153958,"threads_started":1,"wal-append.queue_time_us":520657}
I20260812 19:13:36.805671 24049 consensus_queue.cc:1035] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.141s
I20260812 19:13:36.821789 23764 consensus_queue.cc:1046] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.117s
I20260812 19:13:36.911845 23868 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:13:36.936003 23985 consensus_meta.cc:287] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.918s user 0.018s sys 0.000s
I20260812 19:13:36.957720 21237 raft_consensus.cc:1226] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:36.849924 24053 consensus_queue.cc:1046] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.088s
I20260812 19:13:36.858655 23727 consensus_queue.cc:1046] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.097s
I20260812 19:13:36.937901 24011 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:36.970880 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 154495us
I20260812 19:13:36.948437 22045 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEARNER]: Advancing to term 6
W20260812 19:13:36.973395 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 166908us
I20260812 19:13:36.958722 21036 catalog_manager.cc:5495] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:36.978114 21036 catalog_manager.cc:5495] T b730f70f25e440e68595cc248b40ee98 P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 2 to 3, leader changed from 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:36.854422 24052 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:36.983018 21036 catalog_manager.cc:5495] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 reported cstate change: term changed from 1 to 2, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 2 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } }
I20260812 19:13:36.855913 23961 consensus_queue.cc:237] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:36.944389 23578 consensus_queue.cc:1046] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.221s
I20260812 19:13:36.987329 24034 consensus_queue.cc:1035] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.024s
W20260812 19:13:37.029598 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 176687us
W20260812 19:13:37.005720 23954 consensus_meta.cc:287] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.661s user 0.012s sys 0.002s
I20260812 19:13:36.986253 24053 consensus_queue.cc:1225] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer 93578836968649f6bfcaecc709871412 log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260812 19:13:36.984499 24006 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:36.983896 24059 raft_consensus.cc:2781] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:13:36.991931 23877 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:36.950910 24056 consensus_queue.cc:1035] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.071s
I20260812 19:13:37.040812 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34764 (request call id 25) took 1301 ms. Trace:
W20260812 19:13:37.034206 23762 consensus_meta.cc:287] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.620s user 0.014s sys 0.000s
I20260812 19:13:37.047516 21238 rpcz_store.cc:276] 0812 19:13:35.711662 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:35.740440 (+ 28778us) service_pool.cc:227] Handling call
0812 19:13:37.012316 (+1271876us) spinlock_profiling.cc:243] Waited 90.9 ms on lock 0x7b5400041fd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa0bc3 00007f52de9ea81a 00007f52d80e9d04 00007f52d80e9c92 00007f52d80e9bc1 00007f52d80e9b3c 00007f52d80e8df2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:37.012466 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":209144576}
I20260812 19:13:37.052970 21115 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:37.047763 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 487682us
I20260812 19:13:37.049655 21237 raft_consensus.cc:1226] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260812 19:13:37.059191 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:37.060647 21239 raft_consensus.cc:2379] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:37.068063 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1806) took 1885 ms. Trace:
I20260812 19:13:37.070724 21115 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:37.068768 21318 rpcz_store.cc:276] 0812 19:13:35.182774 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:36.014796 (+832022us) service_pool.cc:227] Handling call
0812 19:13:36.046116 (+ 31320us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:36.062071 (+ 15955us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:36.062109 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:36.115586 (+ 53477us) log.cc:842] Serialized 26 byte log entry
0812 19:13:36.540371 (+424785us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:36.540413 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:36.540488 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:37.067695 (+527207us) raft_consensus.cc:1671] finished
0812 19:13:37.067729 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:37.067968 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1856,"raft.queue_time_us":11464,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":43024,"thread_start_us":294403,"threads_started":1,"wal-append.queue_time_us":372500}
W20260812 19:13:37.073248 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 63281us
I20260812 19:13:37.044510 23578 consensus_queue.cc:1225] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer 63db10063a7d4eb2a9cc9dc952654706 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260812 19:13:37.045234 23954 raft_consensus.cc:2726] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:13:37.077410 21319 raft_consensus.cc:1226] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:37.045714 24058 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:36.994928 24011 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:37.066458 24023 log.cc:924] Time spent T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.879s user 0.011s sys 0.007s
I20260812 19:13:37.052229 24070 raft_consensus.cc:3032] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:37.090438 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:37.069134 23762 raft_consensus.cc:2726] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:37.098469 24058 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:37.099442 24070 raft_consensus.cc:729] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:37.074121 24065 raft_consensus.cc:2781] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election won for term 5
W20260812 19:13:37.100591 24051 log.cc:924] Time spent T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.145s user 0.001s sys 0.000s
W20260812 19:13:37.103349 24047 log.cc:924] Time spent T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.177s user 0.000s sys 0.001s
I20260812 19:13:37.104095 24053 consensus_queue.cc:1035] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:37.114706 22055 raft_consensus.cc:2396] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260812 19:13:37.110041 23764 raft_consensus.cc:2781] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:37.118654 24011 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:37.075058 24059 raft_consensus.cc:686] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:37.047258 24006 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:37.137789 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:37.134905 21112 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:37.078804 23877 consensus_queue.cc:1046] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.045s
I20260812 19:13:37.151741 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.152879 21322 raft_consensus.cc:2396] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 7 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260812 19:13:37.133456 23764 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:37.137068 23918 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:37.174270 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:37.177943 21237 raft_consensus.cc:2379] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 4: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:37.176258 21241 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:13:37.137218 24065 raft_consensus.cc:686] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:37.178798 21241 raft_consensus.cc:595] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:13:37.179195 21241 raft_consensus.cc:603] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:13:37.183569 24059 consensus_queue.cc:237] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:37.185524 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:37.186244 21243 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:37.184396 24045 log.cc:924] Time spent T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.257s user 0.000s sys 0.001s
I20260812 19:13:37.201951 22056 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:37.203672 21155 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:13:37.206519 22011 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:37.207090 21158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1807) took 1144 ms. Trace:
I20260812 19:13:37.207770 21158 rpcz_store.cc:276] 0812 19:13:36.062308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:36.062487 (+ 179us) service_pool.cc:227] Handling call
0812 19:13:36.321502 (+259015us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:36.970950 (+649448us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:36.970989 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:36.971038 (+ 49us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:36.971074 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:37.206677 (+235603us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b5400325854. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:37.206751 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:37.206955 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4062,"raft.queue_time_us":95567,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":7826,"spinlock_wait_cycles":648823680,"thread_start_us":347964,"threads_started":1}
I20260812 19:13:37.209821 24060 raft_consensus.cc:2781] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:13:37.210678 24060 raft_consensus.cc:686] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:37.212144 24060 consensus_queue.cc:237] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:37.213832 23879 raft_consensus.cc:2781] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:37.214490 23952 raft_consensus.cc:3032] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:37.215109 23952 raft_consensus.cc:729] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:37.216871 23897 raft_consensus.cc:3032] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:37.217525 23897 raft_consensus.cc:729] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:37.223592 21342 tablet_copy_service.cc:342] P 93578836968649f6bfcaecc709871412: Request end of tablet copy session 63db10063a7d4eb2a9cc9dc952654706-6f1f5b1c2f184b59b8f452b7f8351692 received from {username='slave'} at 127.0.0.1:50042
I20260812 19:13:37.220916 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1555) took 1459 ms. Trace:
I20260812 19:13:37.226526 23952 consensus_queue.cc:260] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:37.224505 21342 tablet_copy_service.cc:433] P 93578836968649f6bfcaecc709871412: ending tablet copy session 63db10063a7d4eb2a9cc9dc952654706-6f1f5b1c2f184b59b8f452b7f8351692 on tablet 6f1f5b1c2f184b59b8f452b7f8351692 with peer 63db10063a7d4eb2a9cc9dc952654706
W20260812 19:13:37.242334 23735 consensus_meta.cc:287] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.702s user 0.005s sys 0.006s
I20260812 19:13:37.222577 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.226240 21321 rpcz_store.cc:276] 0812 19:13:35.761357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:36.052192 (+290835us) service_pool.cc:227] Handling call
0812 19:13:36.801536 (+749344us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b5800271898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52bce 000000000034aea9 00007f52de972a30 00007f52dea387b4 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:36.801706 (+ 170us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:36.801990 (+ 284us) raft_consensus.cc:1520] Early marking committed up to index 7
0812 19:13:37.011990 (+210000us) log.cc:842] Serialized 14 byte log entry
0812 19:13:37.215890 (+203900us) log.cc:842] Serialized 14 byte log entry
0812 19:13:37.217730 (+ 1840us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:37.218007 (+ 277us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:37.218664 (+ 657us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:37.219299 (+ 635us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:37.220419 (+ 1120us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":273911,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":91,"raft.queue_time_us":1254,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":182,"spinlock_wait_cycles":7371648,"thread_start_us":358138,"threads_started":2}
I20260812 19:13:37.240307 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:37.171414 22046 raft_consensus.cc:1226] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260812 19:13:37.171885 23764 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:37.242858 21203 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:37.244078 21200 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:37.188664 24070 consensus_queue.cc:260] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:37.251132 24052 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:37.251791 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:37.221038 23897 consensus_queue.cc:260] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:37.270334 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:37.271325 21158 raft_consensus.cc:1226] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:37.262090 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:37.226305 23720 raft_consensus.cc:1050] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:13:37.261716 21236 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:37.253039 24011 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:37.272816 24072 consensus_queue.cc:1035] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.127s
W20260812 19:13:37.274181 24077 log.cc:924] Time spent T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20260812 19:13:37.275094 24065 consensus_queue.cc:237] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:37.276587 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
W20260812 19:13:37.270267 24029 log.cc:924] Time spent T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260812 19:13:37.278954 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:37.280246 22045 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEARNER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260812 19:13:37.280532 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 79704.1us
I20260812 19:13:37.307906 21233 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:37.280815 24006 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:37.214525 23879 raft_consensus.cc:686] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:37.278224 24058 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:37.306077 21240 raft_consensus.cc:2396] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260812 19:13:37.318791 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:37.347697 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.304527 23897 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:37.339471 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:37.302021 23720 raft_consensus.cc:1054] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Unable to promote non-voter 63db10063a7d4eb2a9cc9dc952654706: Illegal state: Leader has not yet committed an operation in its own term
I20260812 19:13:37.350629 21236 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:37.350821 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.340576 24032 raft_consensus.cc:2781] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
W20260812 19:13:37.354195 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 219387us
I20260812 19:13:37.316506 21243 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:37.260236 23735 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:37.363103 21321 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:37.371855 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:37.338793 24033 raft_consensus.cc:3032] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:37.348222 24070 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:37.378639 22052 raft_consensus.cc:1261] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260812 19:13:37.385340 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:37.386338 21327 raft_consensus.cc:2354] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 3. Current term is 4.
I20260812 19:13:37.360081 23877 raft_consensus.cc:2726] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:37.394469 23578 consensus_queue.cc:1035] T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20260812 19:13:37.397847 24086 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:37.401710 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:37.375298 24032 raft_consensus.cc:686] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:37.312498 24052 consensus_queue.cc:1046] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20260812 19:13:37.404101 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.425733 21329 raft_consensus.cc:1226] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:37.425930 21318 raft_consensus.cc:1261] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260812 19:13:37.436848 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.405385 21237 raft_consensus.cc:2379] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:37.437841 21322 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:37.318418 24011 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:37.361073 24048 consensus_queue.cc:1035] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.043s
I20260812 19:13:37.380227 24033 raft_consensus.cc:729] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:37.341866 24040 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:37.441761 21319 raft_consensus.cc:2354] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 3. Current term is 4.
I20260812 19:13:37.461580 21204 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:37.462543 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.463352 21327 raft_consensus.cc:2454] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:37.463560 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
W20260812 19:13:37.447320 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 186518us
I20260812 19:13:37.386689 23432 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.046s
I20260812 19:13:37.467190 23432 consensus_queue.cc:1225] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer 63db10063a7d4eb2a9cc9dc952654706 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260812 19:13:37.468552 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 360510us
I20260812 19:13:37.431381 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.470911 22056 pending_rounds.cc:76] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Aborting all ops after (but not including) 3
I20260812 19:13:37.471299 22056 pending_rounds.cc:97] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260812 19:13:37.471597 22056 raft_consensus.cc:2866] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEARNER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:37.443143 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.472114 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.472383 21324 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:37.462015 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:13:37.474261 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:13:37.474510 21324 raft_consensus.cc:1261] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 19:13:37.476325 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.478134 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.478883 21324 raft_consensus.cc:2362] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5. Re-sending same reply.
I20260812 19:13:37.480368 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.482554 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.484431 21327 raft_consensus.cc:1261] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260812 19:13:37.486166 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:37.491151 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:37.492419 21112 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:37.498980 21327 raft_consensus.cc:2379] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:13:37.500603 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 110134us
I20260812 19:13:37.502275 21203 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:37.503494 21238 raft_consensus.cc:1226] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:37.497879 21237 raft_consensus.cc:2362] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4. Re-sending same reply.
I20260812 19:13:37.510314 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:37.518544 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.520207 21318 raft_consensus.cc:1261] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260812 19:13:37.521582 21322 raft_consensus.cc:1261] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260812 19:13:37.522459 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:37.523543 21329 raft_consensus.cc:3037] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 5
I20260812 19:13:37.490669 24072 raft_consensus.cc:2726] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:37.520078 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.539755 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:37.541885 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.540478 21237 raft_consensus.cc:2362] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4. Re-sending same reply.
I20260812 19:13:37.543707 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:37.544813 21235 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 6
I20260812 19:13:37.546149 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:37.546947 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 44762.8us
I20260812 19:13:37.548384 21203 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:37.496914 24058 consensus_queue.cc:1035] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260812 19:13:37.531392 24076 raft_consensus.cc:2726] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:37.539183 22047 raft_consensus.cc:2379] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 4: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:37.465781 23879 consensus_queue.cc:237] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:37.566246 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:37.567647 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.444303 24086 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:37.572448 21237 raft_consensus.cc:2454] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:37.554298 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:37.560587 24072 raft_consensus.cc:2726] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:37.594601 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.578663 23432 raft_consensus.cc:2781] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:37.576409 22047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40624 (request call id 442) took 1102 ms. Trace:
I20260812 19:13:37.600004 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:37.584153 23735 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:37.604286 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 43770us
I20260812 19:13:37.607102 24011 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:37.610832 22045 raft_consensus.cc:2379] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:37.598418 23432 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:37.612958 23432 raft_consensus.cc:3037] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:37.542824 24032 consensus_queue.cc:237] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:37.616123 24081 log.cc:924] Time spent T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.176s user 0.000s sys 0.001s
I20260812 19:13:37.591007 24085 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:37.628387 21329 raft_consensus.cc:2454] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:37.629472 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.632072 21203 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:37.626111 21324 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:37.634296 21203 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:37.635501 24040 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:37.651741 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 164758us
I20260812 19:13:37.674886 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.534832 24033 consensus_queue.cc:260] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:37.649248 24059 raft_consensus.cc:2781] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:37.666630 22050 raft_consensus.cc:1261] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260812 19:13:37.711488 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 77314us
I20260812 19:13:37.837476 21203 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:37.712630 21112 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:37.831382 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 383048us
I20260812 19:13:37.832877 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:37.839219 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 126719us
I20260812 19:13:37.601598 22047 rpcz_store.cc:276] 0812 19:13:36.474268 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:36.474527 (+ 259us) service_pool.cc:227] Handling call
0812 19:13:37.463384 (+988857us) spinlock_profiling.cc:243] Waited 7.25 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:13:37.572274 (+108890us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16676096}
I20260812 19:13:37.708956 21324 raft_consensus.cc:2354] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 3. Current term is 4.
W20260812 19:13:37.891479 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 52307.2us
I20260812 19:13:37.825860 23770 raft_consensus.cc:2726] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:37.645735 24085 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:37.825368 24059 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:37.882237 21241 raft_consensus.cc:1226] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:37.856688 24117 raft_consensus.cc:2781] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:37.892377 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:37.896703 22008 leader_election.cc:334] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.609s (SENT)
W20260812 19:13:37.894126 23868 consensus_meta.cc:287] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.912s user 0.014s sys 0.000s
I20260812 19:13:37.882735 24072 raft_consensus.cc:2726] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:13:37.856652 24049 consensus_queue.cc:1035] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.041s
W20260812 19:13:37.936218 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 55924.9us
I20260812 19:13:37.963027 22008 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:37.967360 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:37.967423 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.968778 21324 raft_consensus.cc:2354] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:13:37.970323 21203 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:37.932483 24059 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:37.935293 24117 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:37.947559 23764 consensus_meta.cc:287] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.656s user 0.013s sys 0.001s
I20260812 19:13:37.975435 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:37.976486 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:37.984689 24107 log.cc:924] Time spent T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
W20260812 19:13:37.987361 24096 log.cc:924] Time spent T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.313s user 0.000s sys 0.001s
I20260812 19:13:37.988530 21241 raft_consensus.cc:1261] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260812 19:13:37.987393 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 90716.4us
I20260812 19:13:37.957662 24033 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:38.001832 22008 leader_election.cc:334] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.555s (SENT)
I20260812 19:13:38.026363 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:37.967726 23958 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:38.030350 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 60135.6us
I20260812 19:13:38.027352 21322 raft_consensus.cc:2454] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
I20260812 19:13:38.038045 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.044616 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:38.044970 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 183683us
I20260812 19:13:37.990506 24117 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:37.991473 24056 raft_consensus.cc:2726] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:38.053074 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.071180 21322 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:38.062398 21203 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:38.066227 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 64438us
I20260812 19:13:38.004179 23764 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:38.080578 22008 leader_election.cc:334] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.801s (SENT)
I20260812 19:13:38.054136 23958 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:38.082973 23764 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:38.081264 22008 leader_election.cc:302] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:37.998723 24086 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:37.969950 23868 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:38.092209 21241 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:38.090219 21112 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:38.094101 22011 leader_election.cc:334] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 1.889s (SENT)
I20260812 19:13:38.107391 22011 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
W20260812 19:13:38.108886 22056 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.009s sys 0.006s
W20260812 19:13:38.109414 22056 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.519s user 0.009s sys 0.006s
I20260812 19:13:38.117928 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.118942 21322 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:38.086412 24038 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:38.128039 21156 raft_consensus.cc:1261] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260812 19:13:38.128674 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.128731 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:38.130321 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 49777.2us
W20260812 19:13:38.131424 22008 leader_election.cc:334] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.837s (SENT)
I20260812 19:13:38.228727 22051 raft_consensus.cc:1226] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:38.233223 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:38.266248 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 172181us
W20260812 19:13:38.233768 22055 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40624 (request call id 445) took 1674 ms (client timeout 1766 ms). Trace:
I20260812 19:13:38.130360 21153 raft_consensus.cc:3032] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:38.147164 21239 raft_consensus.cc:2379] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 6: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
W20260812 19:13:38.196509 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 67641us
I20260812 19:13:38.130760 21327 raft_consensus.cc:2454] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:38.267634 21153 raft_consensus.cc:729] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:38.131795 22008 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:38.148072 21241 raft_consensus.cc:1261] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260812 19:13:38.267319 22055 rpcz_store.cc:269] 0812 19:13:36.559775 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:36.559941 (+ 166us) service_pool.cc:227] Handling call
0812 19:13:38.212202 (+1652261us) spinlock_profiling.cc:243] Waited 552 ms on lock 0x7b540064c8d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:38.233667 (+ 21465us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1270525312}
W20260812 19:13:38.242899 21235 consensus_meta.cc:287] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.690s user 0.011s sys 0.003s
I20260812 19:13:38.270656 21235 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:13:38.268955 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.304603 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.306738 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:38.277909 21114 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:38.304121 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:38.115712 23770 raft_consensus.cc:2726] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:38.305953 21153 consensus_queue.cc:260] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:38.315330 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:38.308482 21322 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 6
I20260812 19:13:38.277285 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1560) took 1027 ms. Trace:
I20260812 19:13:38.316108 21235 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
W20260812 19:13:38.259539 24122 log.cc:924] Time spent T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.142s user 0.001s sys 0.000s
W20260812 19:13:38.251346 24094 log.cc:924] Time spent T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.586s user 0.006s sys 0.006s
W20260812 19:13:38.228003 24045 log.cc:924] Time spent T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.065s user 0.000s sys 0.001s
I20260812 19:13:38.312002 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:38.224749 23432 consensus_meta.cc:287] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.611s user 0.010s sys 0.002s
I20260812 19:13:38.312821 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:38.366047 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 845245us
I20260812 19:13:38.316366 21243 rpcz_store.cc:276] 0812 19:13:37.249500 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:38.232788 (+983288us) service_pool.cc:227] Handling call
0812 19:13:38.277213 (+ 44425us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:38.172981 24054 raft_consensus.cc:2781] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:38.114069 24124 raft_consensus.cc:2726] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:38.330364 21153 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:38.277554 24107 log.cc:924] Time spent T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20260812 19:13:38.317791 24135 raft_consensus.cc:2726] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:38.325692 24116 raft_consensus.cc:1050] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:13:38.334245 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:38.394580 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 123720us
I20260812 19:13:38.163959 24038 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:38.327314 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:38.332803 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:38.335173 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:38.336123 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:38.337549 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.410899 22011 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:38.411418 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.411466 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:38.413101 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:38.209836 23762 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:38.222126 24085 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:38.339677 21156 raft_consensus.cc:1226] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:38.249249 24072 consensus_queue.cc:1035] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.085s
I20260812 19:13:38.327060 22047 raft_consensus.cc:1226] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260812 19:13:38.340682 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:38.327035 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:38.323573 23958 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:38.328629 24144 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:38.355297 22046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 634) took 1756 ms. Trace:
I20260812 19:13:38.328154 23868 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:38.328835 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
W20260812 19:13:38.361908 23897 consensus_meta.cc:287] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.986s user 0.014s sys 0.000s
I20260812 19:13:38.350399 24065 raft_consensus.cc:2726] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:13:38.362668 21237 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:38.348407 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:38.364539 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:38.348066 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:38.354574 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:38.357357 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1553) took 1110 ms. Trace:
I20260812 19:13:38.206885 24060 raft_consensus.cc:2726] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:38.371343 21200 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:38.314031 23764 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:38.276954 23735 raft_consensus.cc:2741] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260812 19:13:38.375653 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:38.467959 21236 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
W20260812 19:13:38.334154 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 202757us
I20260812 19:13:38.325575 22045 raft_consensus.cc:1226] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:38.408334 21114 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:38.347008 23720 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:38.413697 21149 raft_consensus.cc:3037] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:38.445343 24070 consensus_meta.cc:287] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.055s user 0.009s sys 0.003s
I20260812 19:13:38.414918 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:38.423084 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 332960us
I20260812 19:13:38.464562 23658 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:38.437578 21153 raft_consensus.cc:1261] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260812 19:13:38.414695 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1580) took 1013 ms. Trace:
I20260812 19:13:38.453209 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:38.442795 21235 raft_consensus.cc:1261] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260812 19:13:38.491701 21158 raft_consensus.cc:2379] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:38.453616 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:13:38.456701 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 45876.2us
I20260812 19:13:38.451123 21241 raft_consensus.cc:1226] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:38.407691 24091 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:38.461742 21151 raft_consensus.cc:2362] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5. Re-sending same reply.
I20260812 19:13:38.463675 21238 rpcz_store.cc:276] 0812 19:13:37.247058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:37.259601 (+ 12543us) service_pool.cc:227] Handling call
0812 19:13:37.281492 (+ 21891us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:37.505487 (+223995us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:38.356908 (+851421us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b5400303754. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:38.357212 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":510065024}
W20260812 19:13:38.438124 24116 raft_consensus.cc:1054] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Unable to promote non-voter 63db10063a7d4eb2a9cc9dc952654706: Illegal state: Leader has not yet committed an operation in its own term
W20260812 19:13:38.469396 22008 leader_election.cc:334] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.988s (SENT)
I20260812 19:13:38.468321 22053 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 440) took 2016 ms. Trace:
I20260812 19:13:38.475082 21243 raft_consensus.cc:1261] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260812 19:13:38.470839 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 99554.2us
I20260812 19:13:38.477344 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:38.460048 22050 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:38.467124 22048 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEARNER]: Advancing to term 5
I20260812 19:13:38.479823 23897 raft_consensus.cc:2726] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:38.461580 24144 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:38.478200 23868 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:38.473443 24056 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:38.483050 21156 raft_consensus.cc:1261] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260812 19:13:38.486760 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.368994 24054 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:38.488381 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 80172.1us
I20260812 19:13:38.369333 23432 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:38.439328 23762 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:38.492494 22011 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:38.556492 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.539445 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1828) took 1018 ms. Trace:
I20260812 19:13:38.544554 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:38.492530 21152 raft_consensus.cc:2396] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260812 19:13:38.636149 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1547) took 1557 ms. Trace:
W20260812 19:13:38.632436 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 163090us
I20260812 19:13:38.639470 21239 rpcz_store.cc:276] 0812 19:13:37.078963 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:37.079931 (+ 968us) service_pool.cc:227] Handling call
0812 19:13:38.635945 (+1556014us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54003bea54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:38.636086 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2388644608}
I20260812 19:13:38.482600 23952 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:38.638247 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 145828us
I20260812 19:13:38.642869 23958 raft_consensus.cc:2781] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:38.631899 21320 rpcz_store.cc:276] 0812 19:13:37.401467 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:37.401645 (+ 178us) service_pool.cc:227] Handling call
0812 19:13:37.401961 (+ 316us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:37.402194 (+ 233us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:13:37.443977 (+ 41783us) log.cc:842] Serialized 14 byte log entry
0812 19:13:38.362268 (+918291us) spinlock_profiling.cc:243] Waited 5.67 ms on lock 0x7b180019b100. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b014ca 00007f52e01d6f99 00007f52d69c5990 00007f52d6b87864 00007f52d6b87155 00007f52d9a9e3d0 00007f52d9ab0576 00007f52d9ac10d7 00007f52d9ac1058
0812 19:13:38.414164 (+ 51896us) log.cc:842] Serialized 14 byte log entry
0812 19:13:38.414343 (+ 179us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:38.414394 (+ 51us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:38.414451 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:38.414515 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:38.414640 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49062,"prepare.queue_time_us":44029,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":84,"raft.queue_time_us":990657,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"spinlock_wait_cycles":25630720,"thread_start_us":51365,"threads_started":1}
I20260812 19:13:38.645253 24116 consensus_queue.cc:1035] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:13:38.646405 21157 raft_consensus.cc:2354] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5.
I20260812 19:13:38.646402 21156 raft_consensus.cc:1226] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:38.647306 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.483345 24152 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:38.663726 21329 raft_consensus.cc:2379] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:38.639086 21327 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:38.550248 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 58268.6us
I20260812 19:13:38.655403 22011 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:38.669418 24060 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:38.656615 21158 raft_consensus.cc:1226] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260812 19:13:38.670300 23827 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:38.645488 21149 raft_consensus.cc:2454] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:38.670948 23827 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:38.633796 22053 rpcz_store.cc:276] 0812 19:13:36.452183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:36.452369 (+ 186us) service_pool.cc:227] Handling call
0812 19:13:36.461201 (+ 8832us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:36.468683 (+ 7482us) raft_consensus.cc:1520] Early marking committed up to index 7
0812 19:13:36.565912 (+ 97229us) log.cc:842] Serialized 14 byte log entry
0812 19:13:38.364648 (+1798736us) spinlock_profiling.cc:243] Waited 29.9 ms on lock 0x7b180018de40. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d973582c 00007f52d6b6c9dc 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b89e22 00007f52d97275bb 00007f52d972d063 00007f52d972d57b
0812 19:13:38.365107 (+ 459us) log.cc:842] Serialized 14 byte log entry
0812 19:13:38.463718 (+ 98611us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:38.463769 (+ 51us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:38.467927 (+ 4158us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:38.468055 (+ 128us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:38.468238 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8136,"prepare.queue_time_us":206447,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":98,"raft.queue_time_us":1274,"raft.run_cpu_time_us":333,"raft.run_wall_time_us":30054,"spinlock_wait_cycles":90795136,"thread_start_us":266008,"threads_started":3,"wal-append.queue_time_us":520069,"wal-append.run_cpu_time_us":2891,"wal-append.run_wall_time_us":1289816}
I20260812 19:13:38.673224 23827 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:38.675246 22053 raft_consensus.cc:1261] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:38.675407 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:38.676923 21238 raft_consensus.cc:2379] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:38.657686 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:38.679431 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.681260 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.682142 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.683182 21320 raft_consensus.cc:2354] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:13:38.683652 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:38.684269 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.638469 21318 rpcz_store.cc:276] 0812 19:13:37.520509 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:37.521029 (+ 520us) service_pool.cc:227] Handling call
0812 19:13:37.521352 (+ 323us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:37.521543 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 4
0812 19:13:37.521884 (+ 341us) log.cc:842] Serialized 14 byte log entry
0812 19:13:38.538953 (+1017069us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:38.539029 (+ 76us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:38.539076 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:38.539177 (+ 101us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:38.539347 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":847807,"prepare.queue_time_us":2695,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":3481,"raft.queue_time_us":889469,"raft.run_cpu_time_us":65,"raft.run_wall_time_us":65,"thread_start_us":168873,"threads_started":1}
I20260812 19:13:38.686225 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.621923 21322 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:13:38.650068 24144 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:38.679863 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:38.650524 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.670176 24060 consensus_queue.cc:1035] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:13:38.691548 21324 raft_consensus.cc:1261] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260812 19:13:38.695916 23827 consensus_queue.cc:1035] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 19:13:38.701586 24060 consensus_queue.cc:1035] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260812 19:13:38.688007 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:38.494391 22047 raft_consensus.cc:1261] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260812 19:13:38.489210 24135 raft_consensus.cc:2781] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:38.490901 24085 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:38.475116 23764 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:38.483805 24117 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:38.655565 23958 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:39.013625 23958 raft_consensus.cc:3037] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:38.710860 22008 leader_election.cc:334] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.194s (SENT)
I20260812 19:13:39.019585 22008 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:38.507905 23658 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:38.489230 23720 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:39.020571 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:38.636053 24054 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:38.977208 21324 raft_consensus.cc:1261] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260812 19:13:38.747216 21155 raft_consensus.cc:1261] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260812 19:13:39.012800 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:39.023593 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 312709us
I20260812 19:13:38.458477 22046 rpcz_store.cc:276] 0812 19:13:36.598711 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:36.637423 (+ 38712us) service_pool.cc:227] Handling call
0812 19:13:37.134181 (+496758us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:37.269937 (+135756us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:38.355000 (+1085063us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b540061fb50. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:38.355218 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1130299264}
I20260812 19:13:38.579753 23425 raft_consensus.cc:3032] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
W20260812 19:13:39.026356 21240 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1531) took 2452 ms (client timeout 1916 ms). Trace:
W20260812 19:13:39.048221 21240 rpcz_store.cc:269] 0812 19:13:36.574291 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:36.574520 (+ 229us) service_pool.cc:227] Handling call
0812 19:13:39.026055 (+2451535us) spinlock_profiling.cc:243] Waited 703 ms on lock 0x7b54001f9854. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:39.026273 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1616929536}
W20260812 19:13:38.980769 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 108866us
I20260812 19:13:38.982784 21318 raft_consensus.cc:1261] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260812 19:13:39.055163 22011 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:39.063963 21327 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:39.112422 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:39.113186 21327 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 6
W20260812 19:13:38.858589 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 220069us
I20260812 19:13:39.066497 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:39.067133 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:39.128609 21152 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:13:38.860695 24092 log.cc:924] Time spent T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 1.173s user 0.000s sys 0.011s
W20260812 19:13:38.858245 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 220137us
I20260812 19:13:39.057884 21320 raft_consensus.cc:2454] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
W20260812 19:13:38.860204 24059 consensus_meta.cc:287] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.835s user 0.012s sys 0.001s
I20260812 19:13:39.003845 24072 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:39.076718 21154 raft_consensus.cc:2362] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5. Re-sending same reply.
I20260812 19:13:39.088937 21244 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260812 19:13:38.965319 24038 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:39.063313 24085 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:39.095800 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:39.103938 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:39.052170 23425 raft_consensus.cc:729] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:39.108039 21156 raft_consensus.cc:1261] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260812 19:13:38.826896 24060 consensus_queue.cc:1035] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.337s
I20260812 19:13:39.161423 21200 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:39.119742 23827 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:38.652539 24056 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:39.177821 21155 raft_consensus.cc:2396] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260812 19:13:38.653206 24091 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:39.188650 21156 pending_rounds.cc:76] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059: Aborting all ops after (but not including) 2
I20260812 19:13:39.189177 21156 pending_rounds.cc:97] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260812 19:13:39.189519 21156 raft_consensus.cc:2866] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:39.189966 21156 pending_rounds.cc:97] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260812 19:13:39.182170 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:39.190296 21156 raft_consensus.cc:2866] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:39.185927 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:39.178196 21114 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:39.164405 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:39.193853 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:39.191926 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 138296us
I20260812 19:13:39.197728 21324 raft_consensus.cc:2454] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:38.825870 24152 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:39.188521 24060 consensus_queue.cc:1225] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer f741102bfe1441d1b453e5a838649059 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260812 19:13:39.209884 21320 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:39.210407 21324 raft_consensus.cc:1261] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260812 19:13:39.114063 24147 consensus_queue.cc:1046] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.119s
W20260812 19:13:39.219527 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 58230.9us
W20260812 19:13:39.205665 22045 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.093s sys 0.146s
W20260812 19:13:39.241722 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 102675us
I20260812 19:13:39.243599 21114 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:39.241858 24060 raft_consensus.cc:2726] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:39.242322 24142 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:39.297312 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 242236us
I20260812 19:13:39.299466 21327 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:39.179785 24072 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:39.243450 21329 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
W20260812 19:13:39.306001 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 62493.9us
I20260812 19:13:39.210820 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:39.275882 21320 raft_consensus.cc:1261] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260812 19:13:39.207477 23762 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:39.267916 21200 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:39.207358 24140 raft_consensus.cc:2726] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260812 19:13:39.243048 22045 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s user 0.093s sys 0.151s
I20260812 19:13:39.297672 21324 raft_consensus.cc:1261] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260812 19:13:39.191138 23827 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:39.235486 24116 raft_consensus.cc:3037] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:39.308416 21318 raft_consensus.cc:2454] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
W20260812 19:13:39.310560 22055 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.826s user 0.014s sys 0.011s
W20260812 19:13:39.325527 22055 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.842s user 0.015s sys 0.011s
I20260812 19:13:39.302241 21327 raft_consensus.cc:1261] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260812 19:13:39.297212 23985 raft_consensus.cc:2726] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:39.316696 24117 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:39.350023 23939 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260812 19:13:39.249029 24059 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:39.327841 23658 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:39.328240 23764 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:39.348585 23720 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:39.314651 24147 consensus_queue.cc:1035] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260812 19:13:39.378136 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:39.311561 24142 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:39.383352 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 142653us
I20260812 19:13:39.365134 24130 raft_consensus.cc:2726] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:39.380167 21327 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:39.374698 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:39.393532 21320 raft_consensus.cc:2396] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260812 19:13:39.399690 23764 consensus_queue.cc:1035] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260812 19:13:39.402815 21200 leader_election.cc:302] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:39.399762 24168 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:39.414440 24168 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:39.415169 24155 log.cc:924] Time spent T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
I20260812 19:13:39.416535 21150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1840) took 1094 ms. Trace:
I20260812 19:13:39.416953 21150 rpcz_store.cc:276] 0812 19:13:38.322472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:38.482273 (+159801us) service_pool.cc:227] Handling call
0812 19:13:38.482679 (+ 406us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:38.483083 (+ 404us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:38.483122 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:38.483724 (+ 602us) log.cc:842] Serialized 26 byte log entry
0812 19:13:39.011745 (+528021us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:39.011787 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:39.011860 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:39.416294 (+404434us) raft_consensus.cc:1671] finished
0812 19:13:39.416313 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:39.416468 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13945,"raft.queue_time_us":1285,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":4258,"thread_start_us":304431,"threads_started":1,"wal-append.queue_time_us":724160}
W20260812 19:13:39.417492 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 217503us
I20260812 19:13:39.365394 24164 consensus_queue.cc:1035] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260812 19:13:39.403425 24085 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:39.400427 24166 consensus_queue.cc:1035] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
W20260812 19:13:39.531961 21153 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.010s sys 0.051s
W20260812 19:13:39.590489 21115 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.050s sys 0.051s
I20260812 19:13:39.518095 21150 raft_consensus.cc:1226] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
W20260812 19:13:39.753711 21153 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.725s user 0.040s sys 0.066s
I20260812 19:13:39.444657 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:39.745317 22008 leader_election.cc:334] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.405s (SENT)
W20260812 19:13:39.729909 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 422173us
W20260812 19:13:39.758829 21115 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.683s user 0.055s sys 0.086s
W20260812 19:13:39.761097 22008 leader_election.cc:334] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.598s (SENT)
W20260812 19:13:39.768705 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 706290us
I20260812 19:13:39.768957 22008 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:39.443537 21327 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:39.760026 21320 raft_consensus.cc:2454] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
W20260812 19:13:39.740238 23952 consensus_meta.cc:287] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.062s user 0.012s sys 0.005s
W20260812 19:13:39.437986 24075 log.cc:924] Time spent T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
I20260812 19:13:39.411072 24135 raft_consensus.cc:686] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:39.627519 24163 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:39.782514 21152 consensus_meta.cc:287] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.616s user 0.006s sys 0.005s
I20260812 19:13:39.437559 24091 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:39.676514 24079 raft_consensus.cc:3032] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:39.758256 24170 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:39.703516 24072 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:39.449148 23720 raft_consensus.cc:2726] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:39.825251 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:39.843405 21150 raft_consensus.cc:2454] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:39.520931 24152 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:39.466812 23958 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:39.737267 24059 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:39.828133 21152 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:39.857713 21152 pending_rounds.cc:76] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059: Aborting all ops after (but not including) 4
W20260812 19:13:39.700851 24156 log.cc:924] Time spent T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.391s user 0.001s sys 0.001s
I20260812 19:13:39.701927 24168 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:39.844972 22011 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:39.821419 24124 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:39.823935 21112 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:39.848584 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 87549.9us
I20260812 19:13:39.830127 24079 raft_consensus.cc:729] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:39.860304 21152 pending_rounds.cc:97] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20260812 19:13:39.869843 21152 raft_consensus.cc:2866] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260812 19:13:39.866056 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 463304us
I20260812 19:13:39.871156 21200 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:39.869369 22045 raft_consensus.cc:1226] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5. Ops: []
W20260812 19:13:39.876020 22048 consensus_meta.cc:287] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.087s user 0.014s sys 0.000s
I20260812 19:13:39.879495 24179 raft_consensus.cc:2781] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:39.881780 23958 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:39.882293 21153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1841) took 1559 ms. Trace:
I20260812 19:13:39.862095 24053 raft_consensus.cc:2726] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:39.888792 21153 rpcz_store.cc:276] 0812 19:13:38.322908 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:38.535744 (+212836us) service_pool.cc:227] Handling call
0812 19:13:38.536208 (+ 464us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:38.632933 (+ 96725us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:38.632981 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:38.655317 (+ 22336us) log.cc:842] Serialized 26 byte log entry
0812 19:13:39.759381 (+1104064us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:39.759419 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:39.759482 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:39.881942 (+122460us) raft_consensus.cc:1671] finished
0812 19:13:39.881973 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:39.882194 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9433,"raft.queue_time_us":24392,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":35204,"thread_start_us":730536,"threads_started":1,"wal-append.queue_time_us":898041}
I20260812 19:13:39.865187 24152 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:39.883109 22048 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEARNER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:39.899358 24079 consensus_queue.cc:260] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:39.901361 21149 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1848) took 1193 ms. Trace:
W20260812 19:13:39.902946 22050 consensus_meta.cc:287] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.165s user 0.005s sys 0.009s
I20260812 19:13:39.868885 23720 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:39.876299 24177 consensus_queue.cc:1035] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260812 19:13:39.680003 24167 consensus_queue.cc:1035] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.325s
W20260812 19:13:39.926463 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 57428.2us
I20260812 19:13:39.886837 23697 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:39.921684 21149 rpcz_store.cc:276] 0812 19:13:38.707612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:39.012333 (+304721us) service_pool.cc:227] Handling call
0812 19:13:39.012732 (+ 399us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:39.051969 (+ 39237us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:39.051998 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:39.078053 (+ 26055us) log.cc:842] Serialized 52 byte log entry
0812 19:13:39.437048 (+358995us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:39.507209 (+ 70161us) log.cc:842] Serialized 14 byte log entry
0812 19:13:39.507511 (+ 302us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:39.507625 (+ 114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:39.901089 (+393464us) raft_consensus.cc:1671] finished
0812 19:13:39.901113 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:39.901293 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21710,"prepare.queue_time_us":81497,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":65,"raft.queue_time_us":48833,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":25710,"thread_start_us":296870,"threads_started":1,"wal-append.queue_time_us":757204}
I20260812 19:13:39.840801 24170 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:39.934331 22008 leader_election.cc:334] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 3.147s (SENT)
W20260812 19:13:39.901312 24054 consensus_meta.cc:287] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.834s user 0.007s sys 0.007s
I20260812 19:13:39.887697 24164 raft_consensus.cc:2781] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:39.937423 22008 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:39.890793 24142 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:39.883826 24124 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:39.946292 24158 log.cc:924] Time spent T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1: Append to log took a long time: real 0.661s user 0.000s sys 0.004s
I20260812 19:13:39.838837 24135 consensus_queue.cc:237] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:39.926106 22050 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:39.949976 24164 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:39.958518 22047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 645) took 1001 ms. Trace:
W20260812 19:13:39.959504 24132 log.cc:924] Time spent T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 1.728s user 0.002s sys 0.002s
I20260812 19:13:39.948004 24054 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:39.959538 24169 raft_consensus.cc:2726] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:39.960130 24187 consensus_queue.cc:1035] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.114s
W20260812 19:13:39.961238 24162 log.cc:924] Time spent T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.634s user 0.001s sys 0.000s
I20260812 19:13:39.834662 24163 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:39.931890 22048 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40624 (request call id 456) took 1607 ms (client timeout 2098 ms). Trace:
I20260812 19:13:39.938561 24079 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:39.935489 23658 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:39.987660 22048 rpcz_store.cc:269] 0812 19:13:38.324069 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:38.328534 (+ 4465us) service_pool.cc:227] Handling call
0812 19:13:39.929790 (+1601256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:39.968073 22008 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:39.991611 23658 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:39.967605 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1826) took 2447 ms. Trace:
I20260812 19:13:39.993530 22008 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:39.972333 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:39.971994 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:39.923830 24152 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:39.966581 24168 consensus_queue.cc:1035] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.008s
I20260812 19:13:39.974757 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:39.966677 24157 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:39.977365 24170 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:39.980082 21244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1562) took 1976 ms. Trace:
I20260812 19:13:39.980430 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:39.947489 24011 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:39.982863 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:39.930038 23720 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:39.940392 23697 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:39.970743 24060 raft_consensus.cc:2726] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:39.988667 22056 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 451) took 2519 ms. Trace:
I20260812 19:13:39.989017 22052 raft_consensus.cc:1226] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEARNER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:39.991758 24163 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:39.974864 22050 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40598 (request call id 640) took 1707 ms (client timeout 2109 ms). Trace:
I20260812 19:13:39.980644 24164 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:40.001981 22008 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:39.986580 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 92665.1us
I20260812 19:13:39.993818 21321 rpcz_store.cc:276] 0812 19:13:37.519784 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:37.524856 (+ 5072us) service_pool.cc:227] Handling call
0812 19:13:37.548095 (+ 23239us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:38.489198 (+941103us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:38.489234 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:38.489803 (+ 569us) log.cc:842] Serialized 26 byte log entry
0812 19:13:38.737407 (+247604us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:38.737461 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:38.737567 (+ 106us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:39.967364 (+1229797us) raft_consensus.cc:1671] finished
0812 19:13:39.967387 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:39.967531 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":818597,"raft.queue_time_us":855625,"raft.run_cpu_time_us":61,"raft.run_wall_time_us":62,"thread_start_us":276358,"threads_started":2,"wal-append.queue_time_us":648164}
W20260812 19:13:40.007712 21287 connection.cc:232] RPC call timeout handler was delayed by -7.2840716140344739s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1509s for a response
W20260812 19:13:40.031999 21287 connection.cc:232] RPC call timeout handler was delayed by -7.2840716140344739s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1509s for a response
W20260812 19:13:40.032312 21287 connection.cc:232] RPC call timeout handler was delayed by -7.2822716139489785s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1511s for a response
W20260812 19:13:40.032467 21287 connection.cc:232] RPC call timeout handler was delayed by -7.2687716139480472s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
W20260812 19:13:40.032608 21287 connection.cc:232] RPC call timeout handler was delayed by -7.2687716139480472s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
W20260812 19:13:40.032744 21287 connection.cc:232] RPC call timeout handler was delayed by -7.2003716139588505s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1602s for a response
I20260812 19:13:40.009469 21244 rpcz_store.cc:276] 0812 19:13:38.004144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:38.045784 (+ 41640us) service_pool.cc:227] Handling call
0812 19:13:39.026212 (+980428us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:39.171539 (+145327us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:39.977364 (+805825us) spinlock_profiling.cc:243] Waited 46.7 ms on lock 0x7b54001f9854. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:39.978529 (+ 1165us) spinlock_profiling.cc:243] Waited 980 ms on lock 0x7b54001f9850. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:39.979972 (+ 1443us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2361273600}
W20260812 19:13:40.032941 21287 connection.cc:232] RPC call timeout handler was delayed by -7.2003716139588505s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1602s for a response
W20260812 19:13:40.033229 21287 connection.cc:232] RPC call timeout handler was delayed by -7.19857161398977s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16040000000000001s for a response
W20260812 19:13:40.033499 21287 connection.cc:232] RPC call timeout handler was delayed by -7.1652716139797121s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16410000000000002s for a response
I20260812 19:13:40.000908 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:39.994971 21243 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:39.988783 24054 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:40.019433 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 76252.3us
W20260812 19:13:40.008533 24116 consensus_meta.cc:287] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.654s user 0.015s sys 0.000s
I20260812 19:13:40.016443 22056 rpcz_store.cc:276] 0812 19:13:37.469395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:37.470284 (+ 889us) service_pool.cc:227] Handling call
0812 19:13:37.470647 (+ 363us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:37.554377 (+ 83730us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:37.554413 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:37.555007 (+ 594us) log.cc:842] Serialized 52 byte log entry
0812 19:13:38.109893 (+554886us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:38.109950 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:38.110060 (+ 110us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:39.958708 (+1848648us) spinlock_profiling.cc:243] Waited 29 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:39.988286 (+ 29578us) raft_consensus.cc:1671] finished
0812 19:13:39.988310 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:39.988580 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10104,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":6193,"spinlock_wait_cycles":66726784,"thread_start_us":519302,"threads_started":1,"wal-append.queue_time_us":614919}
I20260812 19:13:40.007951 24179 raft_consensus.cc:686] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:39.998870 24169 raft_consensus.cc:2726] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:40.009331 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:40.044631 21203 leader_election.cc:334] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Timed out: RequestConsensusVote RPC to 127.19.70.129:36551 timed out after 1.913s (SENT)
W20260812 19:13:40.033713 21287 connection.cc:232] RPC call timeout handler was delayed by -7.1292716140160337s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16810000000000003s for a response
I20260812 19:13:40.011595 23958 raft_consensus.cc:2781] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:40.036700 22011 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:40.043956 21200 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:40.053005 21287 connection.cc:232] RPC call timeout handler was delayed by -7.0842716139741242s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17310000000000003s for a response
W20260812 19:13:40.053614 21287 connection.cc:232] RPC call timeout handler was delayed by -7.0581716139568016s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17600000000000002s for a response
I20260812 19:13:40.053918 23952 raft_consensus.cc:2781] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:39.979274 22047 rpcz_store.cc:276] 0812 19:13:38.957649 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:39.024565 (+ 66916us) service_pool.cc:227] Handling call
0812 19:13:39.025133 (+ 568us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:39.228631 (+203498us) raft_consensus.cc:1520] Early marking committed up to index 4
0812 19:13:39.235373 (+ 6742us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:39.350539 (+115166us) log.cc:842] Serialized 26 byte log entry
0812 19:13:39.807326 (+456787us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:39.831408 (+ 24082us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:39.838055 (+ 6647us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:39.958214 (+120159us) raft_consensus.cc:1671] finished
0812 19:13:39.958237 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:39.958434 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":155918,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":5859,"thread_start_us":557192,"threads_started":2,"wal-append.queue_time_us":516380}
W20260812 19:13:40.054103 21287 connection.cc:232] RPC call timeout handler was delayed by -7.0131716140313074s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18100000000000002s for a response
W20260812 19:13:40.063634 21287 connection.cc:232] RPC call timeout handler was delayed by -7.0131716140313074s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18100000000000002s for a response
W20260812 19:13:40.064406 21287 connection.cc:232] RPC call timeout handler was delayed by -6.998771614045836s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1826s for a response
W20260812 19:13:40.065016 21287 connection.cc:232] RPC call timeout handler was delayed by -6.998771614045836s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1826s for a response
W20260812 19:13:40.065583 21287 connection.cc:232] RPC call timeout handler was delayed by -6.9456716140266508s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1885s for a response
W20260812 19:13:40.065958 21287 connection.cc:232] RPC call timeout handler was delayed by -6.9456716140266508s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1885s for a response
W20260812 19:13:40.066255 21287 connection.cc:232] RPC call timeout handler was delayed by -6.9429716139566153s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1888s for a response
I20260812 19:13:40.041446 24163 raft_consensus.cc:2781] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260812 19:13:40.066784 21287 connection.cc:232] RPC call timeout handler was delayed by -6.9051716140238568s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
I20260812 19:13:40.015681 24011 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:40.063668 23952 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
W20260812 19:13:40.069645 21287 connection.cc:232] RPC call timeout handler was delayed by -6.9051716140238568s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
I20260812 19:13:40.042110 24116 raft_consensus.cc:2726] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260812 19:13:40.028795 22050 rpcz_store.cc:269] 0812 19:13:38.267521 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:38.304791 (+ 37270us) service_pool.cc:227] Handling call
0812 19:13:39.974776 (+1669985us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:40.072093 21287 connection.cc:232] RPC call timeout handler was delayed by -6.8916716140229255s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1945s for a response
I20260812 19:13:40.042407 22046 raft_consensus.cc:1226] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6. Ops: []
W20260812 19:13:40.079643 21203 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.70.129:36551 timed out after 1.954s (SENT) [suppressed 48 similar messages]
I20260812 19:13:40.080567 21236 raft_consensus.cc:3037] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:40.054864 24166 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:40.054669 23958 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:40.074515 21287 connection.cc:232] RPC call timeout handler was delayed by -6.8826716139446944s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1955s for a response
I20260812 19:13:40.065410 23762 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:40.043102 22048 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:40.072841 23952 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:40.069202 24187 raft_consensus.cc:2781] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:13:40.070756 24163 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:40.028255 24157 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:40.113052 21287 connection.cc:232] RPC call timeout handler was delayed by -6.8772716140374541s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19610000000000002s for a response
W20260812 19:13:40.127797 21287 connection.cc:232] RPC call timeout handler was delayed by -6.8124716139864177s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2033s for a response
W20260812 19:13:40.128391 21287 connection.cc:232] RPC call timeout handler was delayed by -6.7494716140208766s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21030000000000004s for a response
W20260812 19:13:40.128847 21287 connection.cc:232] RPC call timeout handler was delayed by -6.7494716140208766s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21030000000000004s for a response
W20260812 19:13:40.129276 21287 connection.cc:232] RPC call timeout handler was delayed by -6.702671614009887s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2155s for a response
W20260812 19:13:40.129966 21287 connection.cc:232] RPC call timeout handler was delayed by -6.7017716140253469s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2156s for a response
W20260812 19:13:40.130767 21287 connection.cc:232] RPC call timeout handler was delayed by -6.6684716140152887s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21930000000000002s for a response
W20260812 19:13:40.131605 21287 connection.cc:232] RPC call timeout handler was delayed by -6.16987161396537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2747s for a response
I20260812 19:13:40.117784 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:40.132896 21327 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:40.082278 21204 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412
W20260812 19:13:40.134672 21287 connection.cc:232] RPC call timeout handler was delayed by -6.0663716139970347s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2862s for a response
W20260812 19:13:40.143034 21287 connection.cc:232] RPC call timeout handler was delayed by -6.0663716139970347s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2862s for a response
W20260812 19:13:40.144037 21287 connection.cc:232] RPC call timeout handler was delayed by -5.8755716140149161s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3074s for a response
I20260812 19:13:40.079921 24177 consensus_queue.cc:1035] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.059s
I20260812 19:13:40.108976 24166 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:40.144850 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:40.187052 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1599) took 1488 ms. Trace:
I20260812 19:13:40.188913 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:40.187682 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:40.192931 21320 raft_consensus.cc:2379] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:40.191783 21322 rpcz_store.cc:276] 0812 19:13:38.698233 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:38.780374 (+ 82141us) service_pool.cc:227] Handling call
0812 19:13:38.780867 (+ 493us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:39.011530 (+230663us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:39.011559 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:39.012098 (+ 539us) log.cc:842] Serialized 26 byte log entry
0812 19:13:39.089732 (+ 77634us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:39.089773 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:39.089842 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:39.960387 (+870545us) spinlock_profiling.cc:243] Waited 3.3 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:40.069507 (+109120us) raft_consensus.cc:1671] finished
0812 19:13:40.069527 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:40.186955 (+117428us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":45329,"raft.run_cpu_time_us":320,"raft.run_wall_time_us":108610,"spinlock_wait_cycles":7588736,"thread_start_us":77194,"threads_started":1,"wal-append.queue_time_us":231148}
I20260812 19:13:40.187448 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:40.107973 23958 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:40.194033 21287 connection.cc:232] RPC call timeout handler was delayed by -5.8755716140149161s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3074s for a response
W20260812 19:13:40.194772 21287 connection.cc:232] RPC call timeout handler was delayed by -5.7999716140329838s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3158s for a response
W20260812 19:13:40.195782 21287 connection.cc:232] RPC call timeout handler was delayed by -5.7999716140329838s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3158s for a response
I20260812 19:13:40.195590 24179 consensus_queue.cc:237] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:40.198195 21287 leader_election.cc:398] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 4.
I20260812 19:13:40.201737 21287 leader_election.cc:401] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:13:40.142517 24187 raft_consensus.cc:686] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:13:40.182560 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 100381us
I20260812 19:13:40.138253 24163 raft_consensus.cc:3037] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:40.215176 21203 leader_election.cc:334] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Timed out: RequestConsensusVote RPC to 127.19.70.129:36551 timed out after 2.008s (SENT)
I20260812 19:13:40.191068 24142 raft_consensus.cc:2781] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:13:40.239045 24142 raft_consensus.cc:686] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:40.240379 24091 raft_consensus.cc:2726] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 4.
I20260812 19:13:40.139227 24176 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:40.241710 24176 raft_consensus.cc:504] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:40.217667 21204 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:40.772835 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 555226us
I20260812 19:13:40.186590 24195 consensus_queue.cc:1035] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.170s
I20260812 19:13:40.186901 24196 consensus_queue.cc:1035] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.123s
I20260812 19:13:40.111932 23762 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:40.142885 23658 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:40.242565 24011 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:40.120122 24116 consensus_queue.cc:1035] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.097s
I20260812 19:13:40.221084 24157 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:40.318698 24152 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:40.240329 24142 consensus_queue.cc:237] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:40.343138 24166 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:40.309362 24169 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:41.031126 24169 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:40.349208 24176 leader_election.cc:288] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:40.188364 24183 log.cc:924] Time spent T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.154s user 0.000s sys 0.001s
W20260812 19:13:40.342064 24190 log.cc:924] Time spent T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.192s user 0.010s sys 0.020s
I20260812 19:13:41.114094 21156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1850) took 2100 ms. Trace:
I20260812 19:13:40.370983 24187 consensus_queue.cc:237] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:40.487411 21233 raft_consensus.cc:1226] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:41.119240 21233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1559) took 3870 ms. Trace:
I20260812 19:13:41.119660 21233 rpcz_store.cc:276] 0812 19:13:37.249215 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:40.486715 (+3237500us) service_pool.cc:227] Handling call
0812 19:13:40.487181 (+ 466us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:41.118933 (+631752us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:41.119169 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:41.120996 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:40.605064 24079 consensus_meta.cc:287] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.617s user 0.014s sys 0.010s
W20260812 19:13:40.637687 24164 consensus_meta.cc:287] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.597s user 0.016s sys 0.000s
I20260812 19:13:40.675127 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:41.123802 21244 raft_consensus.cc:2379] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:41.123440 24164 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:41.124786 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:41.125908 24164 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:40.675341 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:40.694670 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:40.702800 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:40.732007 21287 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.011s sys 0.017s
W20260812 19:13:40.748221 21237 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57682 (request call id 235) took 3354 ms (client timeout 2924 ms). Trace:
I20260812 19:13:40.749701 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1585) took 3026 ms. Trace:
W20260812 19:13:40.754918 24110 log.cc:924] Time spent T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 3.102s user 0.019s sys 0.007s
I20260812 19:13:40.756426 23425 consensus_queue.cc:260] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:40.769683 23961 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:40.780929 23762 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:40.783733 24195 raft_consensus.cc:2781] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:40.787436 24041 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:40.797899 23958 consensus_meta.cc:287] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.576s user 0.009s sys 0.001s
I20260812 19:13:40.318274 23697 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:40.822120 21324 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.007s sys 0.010s
I20260812 19:13:40.848086 24217 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:40.866532 21154 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1828) took 2554 ms (client timeout 1913 ms). Trace:
I20260812 19:13:40.867657 21151 raft_consensus.cc:1261] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260812 19:13:40.870167 24163 consensus_meta.cc:287] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.558s user 0.009s sys 0.001s
I20260812 19:13:40.875499 24056 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:40.880520 22045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40614 (request call id 455) took 1870 ms. Trace:
W20260812 19:13:40.882217 21329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1598) took 2204 ms (client timeout 2155 ms). Trace:
I20260812 19:13:40.909811 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:40.957702 22051 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 635) took 4358 ms. Trace:
I20260812 19:13:40.971169 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:40.974035 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 754588us
W20260812 19:13:40.994884 24135 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.686s user 0.004s sys 0.004s
I20260812 19:13:40.995970 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1576) took 3596 ms. Trace:
W20260812 19:13:41.019141 22008 leader_election.cc:334] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.568s (SENT)
I20260812 19:13:41.142916 24163 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:41.143733 21154 rpcz_store.cc:269] 0812 19:13:38.311673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:38.329860 (+ 18187us) service_pool.cc:227] Handling call
0812 19:13:40.866248 (+2536388us) spinlock_profiling.cc:243] Waited 613 ms on lock 0x7b5400214654. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:40.866435 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1410286464}
I20260812 19:13:41.145117 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:41.145660 24163 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:41.146099 23827 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:41.146508 22056 raft_consensus.cc:1226] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:41.147432 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:41.148280 21243 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:13:41.148660 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1827) took 3628 ms. Trace:
I20260812 19:13:41.149569 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:41.035646 24169 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:41.151877 21238 raft_consensus.cc:2396] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260812 19:13:41.152859 21324 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.969s user 0.008s sys 0.011s
I20260812 19:13:41.152974 24217 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:41.153797 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:41.154044 23961 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:41.028810 24035 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:41.155126 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:41.155313 24035 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:41.156060 21238 raft_consensus.cc:2396] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260812 19:13:41.156425 23961 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:41.156996 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:41.149544 21237 rpcz_store.cc:269] 0812 19:13:37.393583 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:37.587799 (+194216us) service_pool.cc:227] Handling call
0812 19:13:40.748041 (+3160242us) spinlock_profiling.cc:243] Waited 762 ms on lock 0x7b54000b4d54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:40.748150 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1751480576}
W20260812 19:13:41.135445 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 1.36563e+06us
I20260812 19:13:41.158104 24195 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:41.146421 21203 leader_election.cc:334] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Timed out: RequestConsensusVote RPC to 127.19.70.129:36551 timed out after 2.069s (SENT)
I20260812 19:13:41.158648 24195 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:41.159075 21203 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:41.150785 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:41.161134 24180 raft_consensus.cc:2726] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:41.045262 21158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 285) took 2373 ms. Trace:
I20260812 19:13:41.047968 22053 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 648) took 1057 ms. Trace:
I20260812 19:13:41.058955 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:41.163600 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:41.164291 21153 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:41.165302 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:41.060382 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:41.165886 21239 raft_consensus.cc:1261] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260812 19:13:41.166826 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:41.167865 21203 leader_election.cc:334] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Timed out: RequestConsensusVote RPC to 127.19.70.129:36551 timed out after 2.152s (SENT)
W20260812 19:13:41.169080 21203 leader_election.cc:334] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551): Timed out: RequestConsensusVote RPC to 127.19.70.129:36551 timed out after 2.196s (SENT)
I20260812 19:13:41.162494 22053 rpcz_store.cc:276] 0812 19:13:39.990754 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:40.029833 (+ 39079us) service_pool.cc:227] Handling call
0812 19:13:40.030208 (+ 375us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:40.030649 (+ 441us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:40.030679 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:40.031101 (+ 422us) log.cc:842] Serialized 26 byte log entry
0812 19:13:41.020197 (+989096us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b18004913e0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b6c345 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b89e22 00007f52d97275bb 00007f52d972d063 00007f52d972d411 00007f52d9a7a5cf
0812 19:13:41.036843 (+ 16646us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:41.036889 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:41.036979 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:41.047539 (+ 10560us) raft_consensus.cc:1671] finished
0812 19:13:41.047575 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:41.047887 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9056,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":8102,"spinlock_wait_cycles":7137024,"thread_start_us":16353,"threads_started":1,"wal-append.queue_time_us":999766}
I20260812 19:13:41.169466 21203 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:41.152312 21319 rpcz_store.cc:276] 0812 19:13:37.723344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:37.832382 (+109038us) service_pool.cc:227] Handling call
0812 19:13:37.863973 (+ 31591us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:38.413304 (+549331us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:38.413336 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:38.449370 (+ 36034us) log.cc:842] Serialized 26 byte log entry
0812 19:13:40.744120 (+2294750us) spinlock_profiling.cc:243] Waited 6.65 ms on lock 0x7b1800318c40. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b6c345 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b89e22 00007f52d97275bb 00007f52d972d063 00007f52d972d411 00007f52d9a7a5cf
0812 19:13:40.749196 (+ 5076us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:40.749256 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:40.749338 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:40.749390 (+ 52us) raft_consensus.cc:1671] finished
0812 19:13:40.749405 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:40.749631 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":556857,"raft.queue_time_us":550904,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":15245,"spinlock_wait_cycles":15295872,"thread_start_us":4871,"threads_started":1,"wal-append.queue_time_us":2295623}
I20260812 19:13:41.171209 24079 raft_consensus.cc:2726] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:41.171831 21158 rpcz_store.cc:276] 0812 19:13:38.671822 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:38.672038 (+ 216us) service_pool.cc:227] Handling call
0812 19:13:38.672511 (+ 473us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:38.673195 (+ 684us) raft_consensus.cc:1520] Early marking committed up to index 7
0812 19:13:39.114152 (+440957us) log.cc:842] Serialized 14 byte log entry
0812 19:13:39.768616 (+654464us) log.cc:842] Serialized 14 byte log entry
0812 19:13:41.044624 (+1276008us) spinlock_profiling.cc:243] Waited 2.6 ms on lock 0x7b180036d4e0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b014ca 00007f52e01d6f99 00007f52d69c5990 00007f52d6b87864 00007f52d6b87155 00007f52d9a9e3d0 00007f52d9ab0576 00007f52d9ac10d7 00007f52d9ac1058
0812 19:13:41.044830 (+ 206us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:41.044886 (+ 56us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:41.044943 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:41.045022 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:41.045186 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5974,"prepare.queue_time_us":604912,"prepare.run_cpu_time_us":1627,"prepare.run_wall_time_us":10630,"raft.queue_time_us":15339,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":43852,"spinlock_wait_cycles":5989632,"thread_start_us":839191,"threads_started":2,"wal-append.queue_time_us":727310,"wal-append.run_cpu_time_us":4826,"wal-append.run_wall_time_us":1144147}
I20260812 19:13:41.149137 21323 rpcz_store.cc:276] 0812 19:13:37.520138 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:37.520298 (+ 160us) service_pool.cc:227] Handling call
0812 19:13:37.520675 (+ 377us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:37.520909 (+ 234us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:37.520957 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:37.521455 (+ 498us) log.cc:842] Serialized 26 byte log entry
0812 19:13:37.560673 (+ 39218us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:37.560763 (+ 90us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:37.560888 (+ 125us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:40.754206 (+3193318us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:41.148323 (+394117us) raft_consensus.cc:1671] finished
0812 19:13:41.148347 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:41.148594 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44672896,"thread_start_us":38723,"threads_started":1,"wal-append.queue_time_us":123288}
I20260812 19:13:41.114527 21156 rpcz_store.cc:276] 0812 19:13:39.013257 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:39.178910 (+165653us) service_pool.cc:227] Handling call
0812 19:13:39.188391 (+ 9481us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:39.196555 (+ 8164us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:39.196596 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:39.197198 (+ 602us) log.cc:842] Serialized 26 byte log entry
0812 19:13:39.753015 (+555817us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:39.753057 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:39.753140 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:41.113829 (+1360689us) raft_consensus.cc:1671] finished
0812 19:13:41.113854 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:41.114019 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":270144,"threads_started":1,"wal-append.queue_time_us":779993}
I20260812 19:13:41.149514 21328 rpcz_store.cc:276] 0812 19:13:37.399415 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:37.399575 (+ 160us) service_pool.cc:227] Handling call
0812 19:13:37.439239 (+ 39664us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:37.512807 (+ 73568us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:37.512855 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:37.513776 (+ 921us) log.cc:842] Serialized 52 byte log entry
0812 19:13:37.575508 (+ 61732us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:37.599081 (+ 23573us) log.cc:842] Serialized 14 byte log entry
0812 19:13:38.412161 (+813080us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:38.412270 (+ 109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:38.412319 (+ 49us) raft_consensus.cc:1671] finished
0812 19:13:38.412332 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:40.995712 (+2583380us) spinlock_profiling.cc:243] Waited 8.2 ms on lock 0x7b5400295ed0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:40.995882 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":836039,"prepare.queue_time_us":23520,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":63,"raft.queue_time_us":1782304,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":223,"spinlock_wait_cycles":18867584,"thread_start_us":82650,"threads_started":2,"wal-append.queue_time_us":181463,"wal-append.run_cpu_time_us":13797,"wal-append.run_wall_time_us":1703195}
I20260812 19:13:41.181226 22011 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:41.062168 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:40.681103 21200 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:41.186086 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 505079us
I20260812 19:13:41.065634 24124 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:41.187309 24048 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:41.187861 24048 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:41.074231 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 249) took 1875 ms. Trace:
I20260812 19:13:41.188287 24048 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:41.076181 24072 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:41.191051 24072 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:41.193234 21203 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:41.078112 24114 log.cc:924] Time spent T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 2.865s user 0.006s sys 0.003s
I20260812 19:13:41.194993 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:41.195878 21238 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:13:41.084699 23432 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:41.102201 22055 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 452) took 2826 ms. Trace:
I20260812 19:13:41.197149 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:41.099983 24149 log.cc:924] Time spent T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 2.470s user 0.004s sys 0.001s
W20260812 19:13:41.105557 21157 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1832) took 2792 ms (client timeout 2196 ms). Trace:
W20260812 19:13:41.198392 21157 rpcz_store.cc:269] 0812 19:13:38.313021 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:38.313163 (+ 142us) service_pool.cc:227] Handling call
0812 19:13:41.105270 (+2792107us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b5400324954. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:41.105459 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":544264576}
W20260812 19:13:41.103493 24178 log.cc:924] Time spent T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 1.404s user 0.000s sys 0.005s
I20260812 19:13:41.124833 24143 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:41.147907 22051 rpcz_store.cc:276] 0812 19:13:36.599114 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:36.703036 (+103922us) service_pool.cc:227] Handling call
0812 19:13:38.212485 (+1509449us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:38.268060 (+ 55575us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:40.957399 (+2689339us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b540064c8d0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:40.957618 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3470244224}
I20260812 19:13:41.199680 24143 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:41.150554 21287 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.931s user 0.012s sys 0.017s
I20260812 19:13:41.202833 21241 raft_consensus.cc:2454] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
W20260812 19:13:41.203238 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 1.00521e+06us
I20260812 19:13:41.204016 21241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57682 (request call id 253) took 1229 ms. Trace:
I20260812 19:13:41.204454 21241 rpcz_store.cc:276] 0812 19:13:39.974259 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:39.974405 (+ 146us) service_pool.cc:227] Handling call
0812 19:13:41.203820 (+1229415us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54003bf954. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:41.203951 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2766909824}
I20260812 19:13:41.205269 21287 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:41.150697 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:41.151023 22008 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:41.207086 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 188013us
I20260812 19:13:41.209311 24124 raft_consensus.cc:2726] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:41.151136 24041 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:41.211252 23697 raft_consensus.cc:2726] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:41.211217 24143 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:41.149730 23958 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:41.213399 21239 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:41.215453 23958 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:41.152032 24135 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.843s user 0.004s sys 0.005s
I20260812 19:13:41.022249 22046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40614 (request call id 458) took 1096 ms. Trace:
I20260812 19:13:41.039947 23720 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:41.183559 24169 raft_consensus.cc:2781] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:41.216336 22046 rpcz_store.cc:276] 0812 19:13:39.925661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:39.928260 (+ 2599us) service_pool.cc:227] Handling call
0812 19:13:39.973480 (+ 45220us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:40.109092 (+135612us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:41.021466 (+912374us) spinlock_profiling.cc:243] Waited 51.7 ms on lock 0x7b5400622fd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:41.021965 (+ 499us) spinlock_profiling.cc:243] Waited 1.92 ms on lock 0x7b5400622fd0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:41.022162 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":123259136}
I20260812 19:13:41.218096 22047 raft_consensus.cc:1226] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:41.216997 24169 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:41.219769 24169 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:41.148872 22045 rpcz_store.cc:276] 0812 19:13:39.009995 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:39.385721 (+375726us) service_pool.cc:227] Handling call
0812 19:13:39.413957 (+ 28236us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:39.873741 (+459784us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:40.880138 (+1006397us) spinlock_profiling.cc:243] Waited 453 ms on lock 0x7b540058d4d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:40.880430 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1042378624}
I20260812 19:13:41.165894 21153 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:41.216673 24041 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:41.122466 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:41.226239 21233 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:41.226539 24148 consensus_queue.cc:1046] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260812 19:13:41.227692 24152 raft_consensus.cc:3032] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:41.223122 23697 consensus_queue.cc:1046] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260812 19:13:41.228157 21151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1843) took 2829 ms. Trace:
I20260812 19:13:41.228294 24152 raft_consensus.cc:729] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:41.228906 23697 raft_consensus.cc:3032] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:41.229561 22052 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 636) took 3725 ms. Trace:
I20260812 19:13:41.229363 24152 consensus_queue.cc:260] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:41.229954 22052 rpcz_store.cc:276] 0812 19:13:37.504013 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:37.504200 (+ 187us) service_pool.cc:227] Handling call
0812 19:13:39.988762 (+2484562us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:40.028846 (+ 40084us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:41.229299 (+1200453us) spinlock_profiling.cc:243] Waited 2.46 s on lock 0x7b5400467150. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:41.229471 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5668829056}
I20260812 19:13:41.231034 24152 raft_consensus.cc:3037] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:41.229377 23697 raft_consensus.cc:729] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:41.235421 22056 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 650) took 1244 ms. Trace:
I20260812 19:13:41.236163 21203 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:41.234829 23697 consensus_queue.cc:260] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:41.235904 22056 rpcz_store.cc:276] 0812 19:13:39.991483 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:40.050701 (+ 59218us) service_pool.cc:227] Handling call
0812 19:13:40.051071 (+ 370us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:41.147048 (+1095977us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:41.235143 (+ 88095us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54006103d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:41.235349 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2518765312}
I20260812 19:13:41.149407 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:41.239918 21154 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
W20260812 19:13:41.239849 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 57242us
I20260812 19:13:41.197021 22055 rpcz_store.cc:276] 0812 19:13:38.275983 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:38.276228 (+ 245us) service_pool.cc:227] Handling call
0812 19:13:38.276638 (+ 410us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:38.452385 (+175747us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:38.452434 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:38.461896 (+ 9462us) log.cc:842] Serialized 26 byte log entry
0812 19:13:39.331868 (+869972us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:39.331933 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:41.101670 (+1769737us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b54005974d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:41.101736 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:41.101807 (+ 71us) raft_consensus.cc:1671] finished
0812 19:13:41.101819 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:41.102126 (+ 307us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4952,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":120,"spinlock_wait_cycles":402938624,"thread_start_us":842275,"threads_started":1,"wal-append.queue_time_us":1217740,"wal-append.run_cpu_time_us":4500,"wal-append.run_wall_time_us":1113416}
W20260812 19:13:40.726271 23952 consensus_meta.cc:287] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.617s user 0.008s sys 0.004s
W20260812 19:13:41.151337 21329 rpcz_store.cc:269] 0812 19:13:38.677400 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:38.680868 (+ 3468us) service_pool.cc:227] Handling call
0812 19:13:40.881945 (+2201077us) spinlock_profiling.cc:243] Waited 561 ms on lock 0x7b540039b2d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:40.882115 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1290921728}
I20260812 19:13:41.126176 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:41.038720 21236 consensus_meta.cc:287] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.954s user 0.014s sys 0.001s
I20260812 19:13:41.156515 24217 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:41.262560 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 50117.3us
I20260812 19:13:41.228652 21151 rpcz_store.cc:276] 0812 19:13:38.399321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:40.705295 (+2305974us) service_pool.cc:227] Handling call
0812 19:13:40.867221 (+161926us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:41.143499 (+276278us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:41.227918 (+ 84419us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b5400214650. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:41.228086 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":371255168}
W20260812 19:13:41.300757 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 60901us
I20260812 19:13:41.188560 21240 rpcz_store.cc:276] 0812 19:13:39.198542 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:39.198722 (+ 180us) service_pool.cc:227] Handling call
0812 19:13:39.199114 (+ 392us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:39.528307 (+329193us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:39.528338 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:39.532336 (+ 3998us) log.cc:842] Serialized 52 byte log entry
0812 19:13:39.858532 (+326196us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:39.983382 (+124850us) log.cc:842] Serialized 14 byte log entry
0812 19:13:39.990593 (+ 7211us) log.cc:842] Serialized 14 byte log entry
0812 19:13:41.073639 (+1083046us) spinlock_profiling.cc:243] Waited 6.84 ms on lock 0x7b540004c020. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9a39fb3 00007f52d9aa6f0f 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c
0812 19:13:41.073698 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:41.073799 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:41.073956 (+ 157us) raft_consensus.cc:1671] finished
0812 19:13:41.073972 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:41.074159 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47035,"prepare.queue_time_us":135791,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":78,"raft.queue_time_us":207781,"raft.run_cpu_time_us":583,"raft.run_wall_time_us":32130,"spinlock_wait_cycles":15732992,"thread_start_us":674817,"threads_started":3,"wal-append.queue_time_us":362586}
I20260812 19:13:41.366729 23750 consensus_queue.cc:1035] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:41.369555 21233 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:13:41.237610 23697 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:41.364019 21153 raft_consensus.cc:2454] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:41.157740 24035 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:41.375234 21200 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:41.371937 21236 raft_consensus.cc:2454] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
W20260812 19:13:41.384742 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 1.56156e+06us
W20260812 19:13:41.386086 21074 debug-util.cc:398] Leaking SignalData structure 0x7b0800033dc0 after lost signal to thread 19742
W20260812 19:13:41.397755 21074 debug-util.cc:398] Leaking SignalData structure 0x7b08003cfd80 after lost signal to thread 21077
W20260812 19:13:41.398146 21074 debug-util.cc:398] Leaking SignalData structure 0x7b08003cfe80 after lost signal to thread 21093
W20260812 19:13:41.398939 21074 debug-util.cc:398] Leaking SignalData structure 0x7b08000fc9a0 after lost signal to thread 21152
W20260812 19:13:41.399631 21074 debug-util.cc:398] Leaking SignalData structure 0x7b08003b9300 after lost signal to thread 21185
I20260812 19:13:41.400609 21112 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:41.386128 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1841) took 1275 ms. Trace:
I20260812 19:13:41.382696 23952 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:41.409907 21287 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:41.408674 21074 debug-util.cc:398] Leaking SignalData structure 0x7b08003b9260 after lost signal to thread 21186
I20260812 19:13:41.368635 24053 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:41.426220 21074 debug-util.cc:398] Leaking SignalData structure 0x7b080033b680 after lost signal to thread 21270
I20260812 19:13:41.427812 21236 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57682 (request call id 255) took 1448 ms. Trace:
I20260812 19:13:41.427954 21235 raft_consensus.cc:1226] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:41.395828 24144 raft_consensus.cc:1050] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:13:41.429992 21236 rpcz_store.cc:276] 0812 19:13:39.979978 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:39.980135 (+ 157us) service_pool.cc:227] Handling call
0812 19:13:41.427557 (+1447422us) spinlock_profiling.cc:243] Waited 70.7 ms on lock 0x7b540018fe54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:41.427743 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":162709888}
W20260812 19:13:41.429069 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 49201.7us
W20260812 19:13:41.433588 21074 debug-util.cc:398] Leaking SignalData structure 0x7b0800280a00 after lost signal to thread 21359
W20260812 19:13:41.435370 21074 debug-util.cc:398] Leaking SignalData structure 0x7b08003e37c0 after lost signal to thread 22084
W20260812 19:13:41.435861 21074 debug-util.cc:398] Leaking SignalData structure 0x7b08003e3700 after lost signal to thread 23578
I20260812 19:13:41.382736 23720 raft_consensus.cc:2726] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:41.407382 21324 rpcz_store.cc:276] 0812 19:13:40.110482 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:40.110677 (+ 195us) service_pool.cc:227] Handling call
0812 19:13:40.139720 (+ 29043us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:40.139942 (+ 222us) raft_consensus.cc:1520] Early marking committed up to index 2
0812 19:13:41.153360 (+1013418us) log.cc:842] Serialized 14 byte log entry
0812 19:13:41.385705 (+232345us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:41.385765 (+ 60us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:41.385817 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:41.385920 (+ 103us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:41.386067 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":80750,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":188,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":159,"thread_start_us":1200886,"threads_started":2,"wal-append.queue_time_us":232260}
W20260812 19:13:41.439640 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 64990us
W20260812 19:13:41.440925 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 1.66933e+06us
I20260812 19:13:41.483264 22011 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:41.368916 24041 consensus_queue.cc:1046] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20260812 19:13:41.490701 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:41.491734 22055 raft_consensus.cc:2379] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:41.494709 24072 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:41.484439 24228 log.cc:924] Time spent T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.094s user 0.002s sys 0.000s
W20260812 19:13:41.482653 24230 log.cc:924] Time spent T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.094s user 0.002s sys 0.000s
I20260812 19:13:41.486229 24167 raft_consensus.cc:1050] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:13:41.486382 22048 raft_consensus.cc:1226] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6. Ops: []
W20260812 19:13:41.511953 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 128086us
I20260812 19:13:41.513489 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:41.514473 21203 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:41.490947 24135 raft_consensus.cc:2781] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:41.441663 24180 consensus_queue.cc:1046] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.068s
I20260812 19:13:41.498854 24157 raft_consensus.cc:2726] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:41.519603 23750 consensus_queue.cc:1035] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:13:41.519876 24188 consensus_queue.cc:1035] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:13:41.520505 24137 raft_consensus.cc:2726] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:41.521221 24041 raft_consensus.cc:1050] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
W20260812 19:13:41.535259 24041 raft_consensus.cc:1054] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Unable to promote non-voter 63db10063a7d4eb2a9cc9dc952654706: Illegal state: Leader has not yet committed an operation in its own term
I20260812 19:13:41.530715 22011 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:41.515586 24167 raft_consensus.cc:1054] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Unable to promote non-voter 63db10063a7d4eb2a9cc9dc952654706: Illegal state: Leader has not yet committed an operation in its own term
I20260812 19:13:41.507357 24048 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:41.531622 21287 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:41.527211 24135 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:41.539042 23961 raft_consensus.cc:2781] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:41.540230 24041 raft_consensus.cc:2726] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:41.541823 21287 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:41.552904 21287 leader_election.cc:398] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6.
I20260812 19:13:41.553251 21287 leader_election.cc:401] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:13:41.536854 24188 consensus_queue.cc:1225] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer f741102bfe1441d1b453e5a838649059 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260812 19:13:41.545223 24120 log.cc:924] Time spent T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 3.318s user 0.018s sys 0.015s
I20260812 19:13:41.533056 24124 consensus_queue.cc:1035] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.023s
I20260812 19:13:41.549635 24157 raft_consensus.cc:2741] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260812 19:13:41.567111 24143 consensus_queue.cc:1046] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260812 19:13:41.548413 24135 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:41.545419 23961 raft_consensus.cc:686] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:41.568673 23952 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:41.555634 24152 raft_consensus.cc:2781] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:41.541170 24169 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:41.578877 24240 raft_consensus.cc:2726] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6.
I20260812 19:13:41.601918 22051 raft_consensus.cc:2379] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:41.603498 24048 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:41.606032 23961 consensus_queue.cc:237] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:41.582999 24041 consensus_queue.cc:1046] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260812 19:13:41.658562 24187 raft_consensus.cc:3032] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:41.659183 24187 raft_consensus.cc:729] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:41.644007 24195 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:41.661190 21243 raft_consensus.cc:2379] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 6: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:41.612448 23855 consensus_queue.cc:1035] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260812 19:13:41.615200 21074 debug-util.cc:398] Leaking SignalData structure 0x7b08003a59a0 after lost signal to thread 23918
I20260812 19:13:41.616240 23918 raft_consensus.cc:2781] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:41.623432 23578 consensus_queue.cc:1046] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 19:13:41.660637 24187 consensus_queue.cc:260] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:41.625185 24157 consensus_queue.cc:1046] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
W20260812 19:13:41.628772 24234 log.cc:924] Time spent T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.141s user 0.003s sys 0.000s
I20260812 19:13:41.665192 24079 raft_consensus.cc:3032] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:41.665416 24157 raft_consensus.cc:3032] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:41.552435 24189 consensus_queue.cc:1046] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260812 19:13:41.666783 24189 raft_consensus.cc:3032] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:13:41.667296 24189 raft_consensus.cc:729] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:13:41.596580 24232 log.cc:924] Time spent T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.108s user 0.000s sys 0.001s
I20260812 19:13:41.602444 24169 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:41.647341 24144 raft_consensus.cc:1054] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Unable to promote non-voter 63db10063a7d4eb2a9cc9dc952654706: Illegal state: Leader has not yet committed an operation in its own term
W20260812 19:13:41.651675 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 53559us
I20260812 19:13:41.591770 24152 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:41.608959 24053 raft_consensus.cc:2726] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:41.625144 21150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 288) took 2570 ms. Trace:
I20260812 19:13:41.665787 24079 raft_consensus.cc:729] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:41.665833 24157 raft_consensus.cc:729] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:41.710206 21150 rpcz_store.cc:276] 0812 19:13:39.054330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:39.850402 (+796072us) service_pool.cc:227] Handling call
0812 19:13:39.865628 (+ 15226us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:41.156609 (+1290981us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:41.156650 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:41.157424 (+ 774us) log.cc:842] Serialized 26 byte log entry
0812 19:13:41.506194 (+348770us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:41.506235 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:41.506310 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:41.624861 (+118551us) raft_consensus.cc:1671] finished
0812 19:13:41.624882 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:41.625047 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1272454,"raft.queue_time_us":1749842,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":130,"thread_start_us":348305,"threads_started":1,"wal-append.queue_time_us":434600}
I20260812 19:13:41.721290 21200 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:41.667304 24187 raft_consensus.cc:3037] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:41.712731 24152 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:41.764578 24195 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:41.763351 24157 consensus_queue.cc:260] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:41.742542 24079 consensus_queue.cc:260] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:41.767266 24157 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:13:41.767795 21036 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48428 (request call id 247) took 17298 ms (client timeout 15000 ms). Trace:
I20260812 19:13:41.768169 24079 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:41.729854 24072 raft_consensus.cc:2726] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 19:13:41.806263 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 176546us
W20260812 19:13:41.823787 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 239685us
I20260812 19:13:41.840066 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:41.824955 21322 raft_consensus.cc:2932] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:13:41.848340 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:41.768292 21036 rpcz_store.cc:269] 0812 19:13:24.470067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:24.489018 (+ 18951us) service_pool.cc:227] Handling call
0812 19:13:41.767681 (+17278663us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0812 19:13:37.584947 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0812 19:13:37.693646 (+108699us) write_op.cc:432] Acquiring schema lock in shared mode
0812 19:13:37.693706 (+ 60us) write_op.cc:435] Acquired schema lock
0812 19:13:37.693734 (+ 28us) tablet.cc:660] Decoding operations
0812 19:13:37.849324 (+155590us) write_op.cc:620] Acquiring the partition lock for write op
0812 19:13:37.849462 (+ 138us) write_op.cc:641] Partition lock acquired for write op
0812 19:13:37.849508 (+ 46us) tablet.cc:683] Acquiring locks for 21 operations
0812 19:13:38.251043 (+401535us) tablet.cc:699] Row locks acquired
0812 19:13:38.271727 (+ 20684us) write_op.cc:260] PREPARE: finished
0812 19:13:38.328671 (+ 56944us) write_op.cc:270] Start()
0812 19:13:38.328889 (+ 218us) write_op.cc:276] Timestamp: P: 1786562018309412 usec, L: 0
0812 19:13:38.328933 (+ 44us) op_driver.cc:331] REPLICATION: starting
0812 19:13:38.332982 (+ 4049us) log.cc:842] Serialized 8379 byte log entry
0812 19:13:38.580380 (+247398us) spinlock_profiling.cc:243] Waited 4.53 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d6b6c5fe 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b89e22 00007f52d97275bb
0812 19:13:40.069021 (+1488641us) op_driver.cc:444] REPLICATION: finished
0812 19:13:40.344052 (+275031us) write_op.cc:301] APPLY: starting
0812 19:13:40.344244 (+ 192us) tablet.cc:1365] starting BulkCheckPresence
0812 19:13:40.344994 (+ 750us) tablet.cc:1368] finished BulkCheckPresence
0812 19:13:40.345064 (+ 70us) tablet.cc:1370] starting ApplyRowOperation cycle
0812 19:13:40.353171 (+ 8107us) tablet.cc:1381] finished ApplyRowOperation cycle
0812 19:13:40.356622 (+ 3451us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=21
0812 19:13:40.356708 (+ 86us) write_op.cc:312] APPLY: finished
0812 19:13:40.358747 (+ 2039us) log.cc:842] Serialized 145 byte log entry
0812 19:13:40.359072 (+ 325us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:13:40.359399 (+ 327us) write_op.cc:454] Released schema lock
0812 19:13:40.359521 (+ 122us) write_op.cc:341] FINISH: Updating metrics
0812 19:13:40.360001 (+ 480us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:13:40.360052 (+ 51us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":40407,"reported_tablets":29,"child_traces":[["op",{"apply.queue_time_us":236756,"apply.run_cpu_time_us":8838,"apply.run_wall_time_us":16732,"num_ops":21,"prepare.queue_time_us":276530,"prepare.run_cpu_time_us":38763,"prepare.run_wall_time_us":1484632,"raft.queue_time_us":76566,"raft.run_cpu_time_us":96104,"raft.run_wall_time_us":251089,"replication_time_us":1739913,"spinlock_wait_cycles":10430336,"thread_start_us":720732,"threads_started":4,"wal-append.queue_time_us":867038,"wal-append.run_cpu_time_us":30640,"wal-append.run_wall_time_us":2159150}]]}
I20260812 19:13:41.849436 21318 raft_consensus.cc:2396] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260812 19:13:41.856225 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 267903us
I20260812 19:13:41.860713 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:41.862099 21319 raft_consensus.cc:1261] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260812 19:13:41.863984 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 98075.7us
I20260812 19:13:41.868628 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:41.869249 21244 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:41.878777 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:41.878976 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:41.879892 21236 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:41.880204 21319 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:41.866453 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:41.905329 21319 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:13:41.906860 21236 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:13:41.908743 22011 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:41.918494 23961 raft_consensus.cc:2781] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:13:41.919739 24169 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:41.933499 24242 log.cc:924] Time spent T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20260812 19:13:41.955861 21244 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:41.955104 23961 raft_consensus.cc:686] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
W20260812 19:13:41.958775 24233 log.cc:924] Time spent T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.530s user 0.000s sys 0.005s
I20260812 19:13:41.959301 23961 consensus_queue.cc:237] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:41.961786 21152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 289) took 2634 ms. Trace:
I20260812 19:13:41.962298 21152 rpcz_store.cc:276] 0812 19:13:39.327650 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:39.857089 (+529439us) service_pool.cc:227] Handling call
0812 19:13:39.857496 (+ 407us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:41.528838 (+1671342us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:41.528874 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:41.529421 (+ 547us) log.cc:842] Serialized 52 byte log entry
0812 19:13:41.681245 (+151824us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:41.681310 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:41.681428 (+ 118us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:41.961496 (+280068us) raft_consensus.cc:1671] finished
0812 19:13:41.961520 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:41.961710 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1533762,"raft.queue_time_us":1784211,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":158,"thread_start_us":259636,"threads_started":2,"wal-append.queue_time_us":286477}
I20260812 19:13:41.970233 21320 raft_consensus.cc:1261] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260812 19:13:41.953850 24169 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:41.972498 24169 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:41.978850 21074 debug-util.cc:398] Leaking SignalData structure 0x7b0800364b60 after lost signal to thread 24085
W20260812 19:13:41.979300 21074 debug-util.cc:398] Leaking SignalData structure 0x7b0800383760 after lost signal to thread 24092
W20260812 19:13:41.979959 21074 debug-util.cc:398] Leaking SignalData structure 0x7b08002e49c0 after lost signal to thread 24147
I20260812 19:13:41.980010 23877 raft_consensus.cc:3032] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Stepping down as leader of term 4
W20260812 19:13:41.980618 21074 debug-util.cc:398] Leaking SignalData structure 0x7b0800374e80 after lost signal to thread 24170
I20260812 19:13:41.980712 23877 raft_consensus.cc:729] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:13:41.981029 21074 debug-util.cc:398] Leaking SignalData structure 0x7b08003dcc60 after lost signal to thread 24182
I20260812 19:13:41.981828 23877 consensus_queue.cc:260] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:41.984328 21200 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:42.004004 22011 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:41.985458 24072 consensus_queue.cc:1035] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260812 19:13:41.989135 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 505600us
I20260812 19:13:41.989301 23751 raft_consensus.cc:3032] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Stepping down as leader of term 2
W20260812 19:13:41.997028 24247 log.cc:924] Time spent T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.196s user 0.000s sys 0.002s
I20260812 19:13:41.980170 21243 raft_consensus.cc:2454] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
W20260812 19:13:42.013940 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 59691us
I20260812 19:13:42.027837 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:42.017513 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:42.028666 21153 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:42.030269 21244 raft_consensus.cc:2396] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260812 19:13:42.034809 22011 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:42.037528 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.042680 22011 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:42.006590 23751 raft_consensus.cc:729] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:42.009303 23918 raft_consensus.cc:686] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:42.025543 23961 raft_consensus.cc:2781] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:42.028105 23750 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election won for term 7
W20260812 19:13:42.065752 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 665182us
I20260812 19:13:42.029481 24109 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:42.076030 21153 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
W20260812 19:13:42.086969 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 46958.7us
I20260812 19:13:42.090744 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:42.092500 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.077167 24157 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:42.009214 23432 consensus_queue.cc:1035] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.061s
I20260812 19:13:42.051191 24124 raft_consensus.cc:2781] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:42.093714 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:41.994285 24189 consensus_queue.cc:260] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:42.051029 23958 raft_consensus.cc:2781] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:42.104285 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.105487 21152 raft_consensus.cc:2354] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6.
I20260812 19:13:42.108333 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.108897 21287 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:42.109362 21154 raft_consensus.cc:2396] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260812 19:13:42.063362 24243 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:42.112258 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.115075 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.115202 21112 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:42.115434 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.118011 21149 raft_consensus.cc:2379] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:42.118206 21150 raft_consensus.cc:2454] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
W20260812 19:13:42.115940 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 110873us
I20260812 19:13:42.118671 23697 raft_consensus.cc:2726] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:42.123126 24189 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:42.092723 24109 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:42.126847 23918 consensus_queue.cc:237] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:42.128252 21157 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:42.134960 23764 raft_consensus.cc:2781] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:42.135516 23764 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:42.136096 23764 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:42.137331 23750 raft_consensus.cc:686] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:42.137583 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:42.138882 23750 consensus_queue.cc:237] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:42.140774 21329 raft_consensus.cc:1226] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 6. Current term is 7. Ops: []
I20260812 19:13:42.089604 24238 raft_consensus.cc:2781] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:42.063678 24195 consensus_queue.cc:1035] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260812 19:13:42.162690 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:42.170511 22008 leader_election.cc:334] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.720s (SENT)
I20260812 19:13:42.173225 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.186115 21114 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:42.216926 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:42.217703 21329 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 5
I20260812 19:13:42.218207 21149 raft_consensus.cc:1226] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: []
W20260812 19:13:42.221814 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 60652.5us
I20260812 19:13:42.148819 24238 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:42.228776 21114 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:42.106945 24157 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:42.180859 23425 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:42.232735 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:13:42.250138 21287 leader_election.cc:398] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 3.
W20260812 19:13:42.223701 24135 consensus_meta.cc:287] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.633s user 0.010s sys 0.008s
I20260812 19:13:42.260313 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:42.267864 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.268909 21114 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:42.268656 21157 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:42.276885 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:42.277801 21321 raft_consensus.cc:3037] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:42.229605 24065 raft_consensus.cc:2726] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:42.281392 21287 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
W20260812 19:13:42.287716 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 232555us
I20260812 19:13:42.231839 24238 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:42.255944 24127 raft_consensus.cc:2726] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:13:42.292097 23827 raft_consensus.cc:2726] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260812 19:13:42.256364 24072 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:42.143077 24243 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:42.313391 21114 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:42.320742 23961 raft_consensus.cc:686] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:42.332185 21287 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:42.339599 23961 consensus_queue.cc:237] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:42.352787 21114 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:42.355616 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:42.388805 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.399995 21329 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:42.308444 23897 consensus_queue.cc:1035] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.065s
I20260812 19:13:42.353478 24109 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:42.394901 24152 consensus_meta.cc:287] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.636s user 0.012s sys 0.007s
I20260812 19:13:42.274907 24135 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:42.411577 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:13:42.417457 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 64808.6us
I20260812 19:13:42.418736 21114 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:42.424995 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:42.405320 24127 raft_consensus.cc:2726] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:42.352473 24072 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:42.432520 21114 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:42.437040 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 144503us
I20260812 19:13:42.441254 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.445992 21158 raft_consensus.cc:1261] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260812 19:13:42.449703 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.450541 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.452205 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.457041 21115 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:42.407634 24124 raft_consensus.cc:686] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:42.460704 21158 raft_consensus.cc:2379] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:42.460597 21157 raft_consensus.cc:2379] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:42.462343 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:42.462538 21149 raft_consensus.cc:3032] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:42.464198 21149 raft_consensus.cc:729] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:42.459107 24124 consensus_queue.cc:237] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:42.469367 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:42.482858 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.483263 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.470745 22011 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:42.480976 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.481761 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:42.486189 21152 raft_consensus.cc:2362] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5. Re-sending same reply.
I20260812 19:13:42.431663 24109 raft_consensus.cc:2726] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260812 19:13:42.501585 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 44694.7us
I20260812 19:13:42.504385 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:42.504487 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:42.505156 21150 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:42.448509 23827 raft_consensus.cc:2726] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:42.488134 21149 consensus_queue.cc:260] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:42.506407 22048 raft_consensus.cc:2379] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 6: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:42.421684 24152 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:42.448107 24129 consensus_queue.cc:1035] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
I20260812 19:13:42.487715 23432 consensus_queue.cc:1046] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260812 19:13:42.527041 21151 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:13:42.503708 24085 raft_consensus.cc:2781] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election won for term 5
W20260812 19:13:42.599382 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 267278us
W20260812 19:13:42.604400 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 98081.8us
I20260812 19:13:42.508103 24269 raft_consensus.cc:2726] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:42.520959 24157 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:42.617933 21151 pending_rounds.cc:76] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059: Aborting all ops after (but not including) 2
I20260812 19:13:42.495805 24127 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.097s
I20260812 19:13:42.634553 21151 pending_rounds.cc:97] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260812 19:13:42.634982 21151 raft_consensus.cc:2866] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260812 19:13:42.633533 24268 log.cc:924] Time spent T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
I20260812 19:13:42.531613 24240 raft_consensus.cc:2726] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:42.638981 24127 consensus_queue.cc:1225] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer d1e1da0b9585460e8c365bf55679d4fa log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260812 19:13:42.639492 24262 log.cc:924] Time spent T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.302s user 0.001s sys 0.000s
I20260812 19:13:42.645761 23877 raft_consensus.cc:3037] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:42.656700 21322 raft_consensus.cc:1226] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 6. Current term is 7. Ops: []
I20260812 19:13:42.659941 21320 raft_consensus.cc:1226] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:42.519021 24147 consensus_queue.cc:1046] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.102s
W20260812 19:13:42.668210 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 45702us
W20260812 19:13:42.634290 24267 log.cc:924] Time spent T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.228s user 0.000s sys 0.001s
W20260812 19:13:42.700431 22008 leader_election.cc:334] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.541s (SENT)
I20260812 19:13:42.665952 23855 raft_consensus.cc:2726] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:42.624917 24243 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:42.666242 24130 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:42.668229 24148 raft_consensus.cc:2726] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:42.688488 24144 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
W20260812 19:13:42.759397 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 63224.3us
I20260812 19:13:42.735023 24269 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:42.767908 24147 consensus_queue.cc:1225] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Peer d1e1da0b9585460e8c365bf55679d4fa log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260812 19:13:42.806851 21203 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:42.762246 24286 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:42.833807 21203 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
W20260812 19:13:42.835816 21204 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.004s sys 0.016s
W20260812 19:13:42.836403 21204 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.579s user 0.004s sys 0.016s
I20260812 19:13:42.837543 21203 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:42.843559 21321 consensus_meta.cc:287] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.560s user 0.010s sys 0.000s
I20260812 19:13:42.762809 24135 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:42.845731 21321 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:42.855756 21203 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:42.857885 21203 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:42.862578 21287 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
W20260812 19:13:42.832499 23958 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.010s sys 0.010s
W20260812 19:13:42.864604 23958 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.705s user 0.010s sys 0.011s
I20260812 19:13:42.865047 23958 raft_consensus.cc:686] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:42.827356 24269 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:42.802814 24072 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:42.868702 21287 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:42.771646 24243 consensus_queue.cc:1035] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.061s
I20260812 19:13:42.761273 24130 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:42.771139 24144 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:42.950939 21233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 268) took 1083 ms. Trace:
I20260812 19:13:42.972523 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:42.956748 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 100217us
W20260812 19:13:42.962497 22008 leader_election.cc:334] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.924s (SENT)
I20260812 19:13:42.835286 24085 raft_consensus.cc:686] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:42.980441 21243 raft_consensus.cc:1261] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260812 19:13:42.959007 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 90395us
I20260812 19:13:42.978348 22008 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:42.853334 24187 raft_consensus.cc:2726] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260812 19:13:42.935242 21319 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.682s user 0.022s sys 0.031s
W20260812 19:13:42.991753 21319 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s user 0.027s sys 0.031s
I20260812 19:13:42.997557 22056 raft_consensus.cc:1261] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260812 19:13:42.985062 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:42.976878 21233 rpcz_store.cc:276] 0812 19:13:41.867048 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:41.867174 (+ 126us) service_pool.cc:227] Handling call
0812 19:13:41.867502 (+ 328us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:41.884469 (+ 16967us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:41.884513 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:41.892016 (+ 7503us) log.cc:842] Serialized 52 byte log entry
0812 19:13:41.971038 (+ 79022us) spinlock_profiling.cc:243] Waited 2.66 ms on lock 0x7b4800190158. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52de94d7f1 00007f52de9448e7 00007f52de94477a 00007f52dea3895f 00007f52dea38627 00007f52dea4cb2c 00007f52dea4cad8 00007f52dea4ca58 00007f52dea4ca13 00007f52dea4bcbc 00007f52dd9cdbe6
0812 19:13:42.051574 (+ 80536us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:42.051617 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:42.053012 (+ 1395us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:42.950549 (+897537us) raft_consensus.cc:1671] finished
0812 19:13:42.950571 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:42.950859 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34632,"raft.queue_time_us":5528,"raft.run_cpu_time_us":2149,"raft.run_wall_time_us":93121,"spinlock_wait_cycles":6108544,"thread_start_us":131358,"threads_started":1,"wal-append.queue_time_us":324801}
W20260812 19:13:42.990482 21200 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.004s sys 0.058s
W20260812 19:13:43.002048 21200 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.744s user 0.004s sys 0.058s
W20260812 19:13:43.002883 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 771469us
I20260812 19:13:42.986189 21239 raft_consensus.cc:2379] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 6: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
W20260812 19:13:42.947273 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 89455.6us
I20260812 19:13:43.005064 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:43.005909 21203 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:42.933769 24180 raft_consensus.cc:2726] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:13:42.866374 23958 consensus_queue.cc:237] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:43.009267 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:43.017663 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:42.830487 24076 consensus_queue.cc:1046] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.121s
I20260812 19:13:43.019793 21243 raft_consensus.cc:2354] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6.
I20260812 19:13:43.019992 21203 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:43.025544 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:43.026889 21237 raft_consensus.cc:2396] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260812 19:13:43.027048 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:42.937602 24196 raft_consensus.cc:2726] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:43.029449 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:43.039130 21243 raft_consensus.cc:3037] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:42.956323 24291 raft_consensus.cc:2726] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:43.037755 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:42.948941 23750 raft_consensus.cc:2726] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:43.057245 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:43.066781 21239 raft_consensus.cc:2354] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5.
I20260812 19:13:43.067847 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:43.067939 21243 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:42.997745 24285 raft_consensus.cc:2726] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:42.947475 24048 raft_consensus.cc:2726] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:43.100904 21239 raft_consensus.cc:2379] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:43.102089 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1870) took 1188 ms. Trace:
I20260812 19:13:42.981281 24289 raft_consensus.cc:2726] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260812 19:13:43.058431 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 43826us
I20260812 19:13:43.099099 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:43.114398 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:43.115748 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:43.115626 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:43.116016 24278 log.cc:924] Time spent T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.158s user 0.000s sys 0.001s
I20260812 19:13:43.116367 24076 raft_consensus.cc:3032] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:43.117740 24298 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:43.119683 21243 raft_consensus.cc:2454] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:43.108301 21319 rpcz_store.cc:276] 0812 19:13:41.914177 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:41.914822 (+ 645us) service_pool.cc:227] Handling call
0812 19:13:41.915278 (+ 456us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:41.915579 (+ 301us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:42.253030 (+337451us) log.cc:842] Serialized 14 byte log entry
0812 19:13:43.101732 (+848702us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:43.101790 (+ 58us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:43.101845 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:43.101917 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:43.102031 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":115164,"prepare.queue_time_us":83638,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"thread_start_us":1038798,"threads_started":2}
I20260812 19:13:43.072995 24287 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:43.134238 21238 raft_consensus.cc:2379] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:42.835937 24286 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:43.124377 21287 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:43.158438 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:43.098009 24196 raft_consensus.cc:2726] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:42.979259 24281 consensus_queue.cc:1035] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.170s
I20260812 19:13:43.168365 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:43.172072 21319 raft_consensus.cc:2379] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:43.112033 23750 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:43.190521 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:43.191215 21238 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:43.191502 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:43.192126 21150 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:43.195591 24289 raft_consensus.cc:2726] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:43.187376 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:43.189728 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:43.197443 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:43.189864 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 65615.2us
I20260812 19:13:43.123266 24148 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:43.225807 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:43.226864 22055 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:43.147715 24076 raft_consensus.cc:729] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:13:43.183271 24254 log.cc:924] Time spent T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20260812 19:13:43.236972 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:43.258181 21233 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:43.187680 24189 consensus_meta.cc:287] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.064s user 0.016s sys 0.000s
I20260812 19:13:43.245836 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:43.240298 21150 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:13:43.252059 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:43.248587 21238 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:43.265688 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:43.247771 21241 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:43.273353 22011 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:43.230366 24243 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:43.225266 24152 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:43.253048 24296 log.cc:924] Time spent T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.492s user 0.001s sys 0.001s
I20260812 19:13:43.288259 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1877) took 1191 ms. Trace:
I20260812 19:13:43.303772 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:43.303972 21287 leader_election.cc:302] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:43.316895 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:43.305186 21324 rpcz_store.cc:276] 0812 19:13:42.096543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.096775 (+ 232us) service_pool.cc:227] Handling call
0812 19:13:42.097276 (+ 501us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:42.097784 (+ 508us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:42.097825 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:42.105709 (+ 7884us) log.cc:842] Serialized 26 byte log entry
0812 19:13:42.399771 (+294062us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:42.399816 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:42.399904 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:43.288016 (+888112us) raft_consensus.cc:1671] finished
0812 19:13:43.288040 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:43.288187 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6414,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":8955,"thread_start_us":242719,"threads_started":1,"wal-append.queue_time_us":679828}
I20260812 19:13:43.268674 24289 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:43.169387 24298 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:43.348443 21150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:13:43.351250 21150 raft_consensus.cc:595] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
W20260812 19:13:43.354717 21152 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.743s user 0.021s sys 0.046s
W20260812 19:13:43.355166 21152 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.744s user 0.021s sys 0.046s
I20260812 19:13:43.289983 24124 raft_consensus.cc:2781] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:13:43.366890 21233 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:43.184360 24287 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:43.352810 24254 log.cc:924] Time spent T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20260812 19:13:43.235592 24148 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:43.353886 24152 raft_consensus.cc:2726] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:43.318856 24243 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:43.358830 24293 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:43.493775 21329 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.882s user 0.013s sys 0.048s
W20260812 19:13:43.548923 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 287368us
W20260812 19:13:43.590514 21203 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.016s sys 0.043s
I20260812 19:13:43.420390 24124 raft_consensus.cc:686] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:43.368391 24144 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:43.606246 21329 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.996s user 0.019s sys 0.051s
I20260812 19:13:43.490367 24130 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:43.688066 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1618) took 1395 ms. Trace:
I20260812 19:13:43.490955 24306 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:43.688470 21329 rpcz_store.cc:276] 0812 19:13:42.292536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.429431 (+136895us) service_pool.cc:227] Handling call
0812 19:13:42.463156 (+ 33725us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:42.463364 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 7
0812 19:13:42.463682 (+ 318us) log.cc:842] Serialized 14 byte log entry
0812 19:13:43.687627 (+1223945us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:43.687693 (+ 66us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:43.687748 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:43.687830 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:43.687997 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1654,"prepare.queue_time_us":4049,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":137123,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":13272,"thread_start_us":1222856,"threads_started":2,"wal-append.queue_time_us":265132}
I20260812 19:13:43.240686 24085 consensus_queue.cc:237] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:43.638877 21203 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.618s user 0.017s sys 0.046s
W20260812 19:13:43.669431 22008 leader_election.cc:334] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 3.021s (SENT)
W20260812 19:13:43.691407 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 671460us
I20260812 19:13:43.691650 22008 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:43.692698 21203 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:43.690735 22051 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.006s sys 0.005s
W20260812 19:13:43.694249 22051 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s user 0.006s sys 0.006s
I20260812 19:13:43.724491 21203 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:43.668107 24169 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.920s user 0.007s sys 0.004s
I20260812 19:13:43.724397 22051 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 663) took 1196 ms. Trace:
I20260812 19:13:43.731666 21287 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:43.731629 22051 rpcz_store.cc:276] 0812 19:13:42.528033 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.528239 (+ 206us) service_pool.cc:227] Handling call
0812 19:13:42.605903 (+ 77664us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:42.606163 (+ 260us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:43.061946 (+455783us) log.cc:842] Serialized 14 byte log entry
0812 19:13:43.721639 (+659693us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:43.723271 (+ 1632us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:43.723559 (+ 288us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:43.723729 (+ 170us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:43.724304 (+ 575us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2111,"prepare.queue_time_us":748999,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":56,"raft.queue_time_us":25251,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":151,"thread_start_us":963007,"threads_started":2}
I20260812 19:13:43.354205 24289 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:43.703102 23764 consensus_meta.cc:287] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.566s user 0.019s sys 0.000s
I20260812 19:13:43.354840 24076 consensus_queue.cc:260] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:43.692763 24272 raft_consensus.cc:2726] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 19:13:43.736312 24169 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.005s user 0.008s sys 0.004s
I20260812 19:13:43.731179 24196 raft_consensus.cc:2726] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260812 19:13:43.901484 21239 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.008s sys 0.017s
W20260812 19:13:43.888103 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 201028us
W20260812 19:13:43.904585 21238 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.003s sys 0.024s
W20260812 19:13:43.911172 21238 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.551s user 0.004s sys 0.024s
W20260812 19:13:43.912516 21239 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.552s user 0.008s sys 0.019s
W20260812 19:13:43.917826 21324 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.006s sys 0.046s
W20260812 19:13:43.918496 21324 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s user 0.007s sys 0.046s
W20260812 19:13:43.916029 21323 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.635s user 0.035s sys 0.063s
W20260812 19:13:43.919270 21323 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.655s user 0.035s sys 0.063s
I20260812 19:13:43.619467 24293 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:43.923816 21287 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:43.902794 24320 log.cc:924] Time spent T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.172s user 0.000s sys 0.001s
I20260812 19:13:43.701303 24299 consensus_queue.cc:1046] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.208s
I20260812 19:13:43.893023 24269 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:43.969009 24295 log.cc:924] Time spent T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.858s user 0.001s sys 0.001s
I20260812 19:13:43.944398 24076 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:43.878260 23764 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:43.971431 24321 log.cc:924] Time spent T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.343s user 0.002s sys 0.002s
I20260812 19:13:43.943536 24286 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:43.727160 24306 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:43.988897 22008 leader_election.cc:334] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.178s (SENT)
I20260812 19:13:44.009079 22008 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:43.995939 24318 log.cc:924] Time spent T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.570s user 0.000s sys 0.002s
W20260812 19:13:43.996321 24317 log.cc:924] Time spent T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.368s user 0.000s sys 0.001s
W20260812 19:13:43.997323 21186 heartbeater.cc:644] Failed to heartbeat to 127.19.70.190:35311 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.19.70.190:35311 timed out after 15.000s (SENT)
W20260812 19:13:44.004671 21115 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.012s sys 0.013s
W20260812 19:13:44.004990 21152 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.005s sys 0.021s
W20260812 19:13:44.015756 22055 consensus_meta.cc:287] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.788s user 0.013s sys 0.001s
W20260812 19:13:44.022529 21115 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.616s user 0.012s sys 0.013s
W20260812 19:13:44.022810 21152 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.663s user 0.005s sys 0.021s
I20260812 19:13:44.022998 22055 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
W20260812 19:13:44.041796 22008 leader_election.cc:334] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.673s (SENT)
I20260812 19:13:44.042740 22008 leader_election.cc:302] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:44.024060 24309 log.cc:924] Time spent T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.809s user 0.000s sys 0.002s
W20260812 19:13:44.060253 23425 consensus_meta.cc:287] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.786s user 0.022s sys 0.000s
W20260812 19:13:44.060942 24273 log.cc:924] Time spent T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.170s user 0.000s sys 0.000s
I20260812 19:13:44.061265 23425 raft_consensus.cc:3032] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:13:44.061869 23425 raft_consensus.cc:729] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:13:44.063185 24328 log.cc:924] Time spent T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
W20260812 19:13:44.064805 24319 log.cc:924] Time spent T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.198s user 0.000s sys 0.001s
W20260812 19:13:44.065747 24284 log.cc:924] Time spent T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.383s user 0.000s sys 0.001s
W20260812 19:13:44.039289 24312 log.cc:924] Time spent T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.670s user 0.000s sys 0.002s
I20260812 19:13:43.943171 24124 consensus_queue.cc:237] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:44.024797 23918 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:44.053610 24286 raft_consensus.cc:2726] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:44.054642 24287 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:44.054419 24243 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:44.128484 24088 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.548s user 0.006s sys 0.014s
I20260812 19:13:44.066972 24298 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:44.067358 24148 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:44.144730 24088 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.629s user 0.006s sys 0.014s
W20260812 19:13:44.141335 24303 log.cc:924] Time spent T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.131s user 0.000s sys 0.002s
I20260812 19:13:44.190193 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1889) took 1042 ms. Trace:
W20260812 19:13:44.153383 23750 consensus_meta.cc:287] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.963s user 0.011s sys 0.001s
W20260812 19:13:44.155239 24238 consensus_meta.cc:287] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.802s user 0.018s sys 0.000s
W20260812 19:13:44.177786 24322 log.cc:924] Time spent T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.456s user 0.000s sys 0.002s
I20260812 19:13:44.162300 24157 raft_consensus.cc:2741] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260812 19:13:44.197871 24167 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.328s user 0.006s sys 0.007s
I20260812 19:13:44.189258 24289 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:44.142167 23918 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:44.221158 24167 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.352s user 0.007s sys 0.007s
W20260812 19:13:44.233692 24239 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.578s user 0.005s sys 0.010s
W20260812 19:13:44.289287 24323 log.cc:924] Time spent T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.175s user 0.000s sys 0.001s
W20260812 19:13:44.290066 24333 log.cc:924] Time spent T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
W20260812 19:13:44.278435 24330 log.cc:924] Time spent T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.276s user 0.002s sys 0.000s
I20260812 19:13:44.236899 24293 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:44.257473 23764 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:44.440658 23764 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:44.384328 21151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1879) took 1931 ms. Trace:
I20260812 19:13:44.386160 21154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1874) took 1942 ms. Trace:
W20260812 19:13:44.386926 21202 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.011s sys 0.010s
W20260812 19:13:44.386926 21200 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.653s user 0.000s sys 0.025s
I20260812 19:13:44.387708 21157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1873) took 1943 ms. Trace:
W20260812 19:13:44.443184 21202 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.704s user 0.011s sys 0.010s
W20260812 19:13:44.443270 21200 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.709s user 0.000s sys 0.026s
I20260812 19:13:44.442077 23764 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:44.444305 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 710321us
I20260812 19:13:44.443419 21157 rpcz_store.cc:276] 0812 19:13:42.443717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.489078 (+ 45361us) service_pool.cc:227] Handling call
0812 19:13:42.489549 (+ 471us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:42.489982 (+ 433us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:42.490022 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:42.509923 (+ 19901us) log.cc:842] Serialized 26 byte log entry
0812 19:13:42.780121 (+270198us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:42.780237 (+ 116us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:42.780386 (+ 149us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:44.387131 (+1606745us) raft_consensus.cc:1671] finished
0812 19:13:44.387180 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.387485 (+ 305us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47803,"raft.queue_time_us":5510,"raft.run_cpu_time_us":537,"raft.run_wall_time_us":111842,"thread_start_us":248565,"threads_started":1,"wal-append.queue_time_us":1439242}
I20260812 19:13:44.442826 21151 rpcz_store.cc:276] 0812 19:13:42.452949 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.616976 (+164027us) service_pool.cc:227] Handling call
0812 19:13:42.617375 (+ 399us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:42.696673 (+ 79298us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:42.696718 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:42.697402 (+ 684us) log.cc:842] Serialized 52 byte log entry
0812 19:13:43.120717 (+423315us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:43.120757 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:43.120828 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:44.383060 (+1262232us) raft_consensus.cc:1671] finished
0812 19:13:44.383175 (+ 115us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.383889 (+ 714us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":103726,"raft.queue_time_us":61938,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":14798,"thread_start_us":356195,"threads_started":1,"wal-append.queue_time_us":851093}
I20260812 19:13:44.449201 21157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1881) took 1995 ms. Trace:
I20260812 19:13:44.449786 21151 raft_consensus.cc:1226] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:44.449743 21157 rpcz_store.cc:276] 0812 19:13:42.453613 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:44.447445 (+1993832us) service_pool.cc:227] Handling call
0812 19:13:44.448267 (+ 822us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:44.448716 (+ 449us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:44.448750 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:44.448835 (+ 85us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:44.448896 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.448970 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.449097 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:44.387842 21203 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.652s user 0.003s sys 0.037s
W20260812 19:13:44.388118 21233 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.647s user 0.012s sys 0.018s
I20260812 19:13:44.450585 21151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1885) took 1995 ms. Trace:
W20260812 19:13:44.451186 21233 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s user 0.012s sys 0.018s
I20260812 19:13:44.442888 21154 rpcz_store.cc:276] 0812 19:13:42.444031 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.491982 (+ 47951us) service_pool.cc:227] Handling call
0812 19:13:42.508953 (+ 16971us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:42.669763 (+160810us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:42.669813 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:42.670492 (+ 679us) log.cc:842] Serialized 26 byte log entry
0812 19:13:43.137624 (+467132us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:43.137672 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:43.137779 (+ 107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:44.385490 (+1247711us) raft_consensus.cc:1671] finished
0812 19:13:44.385577 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.385918 (+ 341us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":136289,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"thread_start_us":565635,"threads_started":2,"wal-append.queue_time_us":642942}
I20260812 19:13:44.452657 23764 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:44.388441 21158 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.459s user 0.017s sys 0.015s
W20260812 19:13:44.454027 21158 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.525s user 0.017s sys 0.016s
I20260812 19:13:44.454420 21154 raft_consensus.cc:1226] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:44.394735 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1594) took 1117 ms. Trace:
W20260812 19:13:44.395617 21241 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.021s sys 0.014s
I20260812 19:13:44.398581 21114 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:44.404744 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 480886us
W20260812 19:13:44.406026 21319 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.208s user 0.006s sys 0.006s
I20260812 19:13:44.407262 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1881) took 1376 ms. Trace:
I20260812 19:13:44.199285 21323 rpcz_store.cc:276] 0812 19:13:43.148139 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:43.224322 (+ 76183us) service_pool.cc:227] Handling call
0812 19:13:43.252359 (+ 28037us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:43.252707 (+ 348us) raft_consensus.cc:1520] Early marking committed up to index 4
0812 19:13:43.263680 (+ 10973us) log.cc:842] Serialized 14 byte log entry
0812 19:13:44.189626 (+925946us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:44.189717 (+ 91us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:44.189799 (+ 82us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.189906 (+ 107us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.190098 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29397,"prepare.queue_time_us":9841,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":79,"raft.queue_time_us":242083,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":895973,"threads_started":2,"wal-append.queue_time_us":789452}
I20260812 19:13:44.410552 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1622) took 1228 ms. Trace:
W20260812 19:13:44.342417 24335 log.cc:924] Time spent T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.051s user 0.000s sys 0.001s
W20260812 19:13:44.423992 22052 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.932s user 0.002s sys 0.010s
I20260812 19:13:44.425384 22053 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 463) took 1661 ms. Trace:
I20260812 19:13:44.147395 23425 consensus_queue.cc:260] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:44.215080 23750 raft_consensus.cc:2726] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 19:13:44.450881 21203 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.723s user 0.004s sys 0.037s
I20260812 19:13:44.451273 21151 rpcz_store.cc:276] 0812 19:13:42.455209 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:44.448798 (+1993589us) service_pool.cc:227] Handling call
0812 19:13:44.449392 (+ 594us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:44.450297 (+ 905us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.450519 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:44.289340 23897 raft_consensus.cc:3032] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:44.455700 21158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1875) took 2011 ms. Trace:
W20260812 19:13:44.388743 21236 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.359s user 0.000s sys 0.011s
W20260812 19:13:44.388456 21204 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.005s sys 0.030s
I20260812 19:13:44.458395 23918 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:44.460428 24053 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:44.469980 24088 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:44.475514 21157 raft_consensus.cc:1226] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
W20260812 19:13:44.630806 21204 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.884s user 0.006s sys 0.031s
I20260812 19:13:44.631743 21157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1888) took 2175 ms. Trace:
I20260812 19:13:44.632352 21157 rpcz_store.cc:276] 0812 19:13:42.456246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:44.471044 (+2014798us) service_pool.cc:227] Handling call
0812 19:13:44.473996 (+ 2952us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:44.631337 (+157341us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.631634 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:44.633788 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 909294us
I20260812 19:13:44.214221 24238 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:44.639858 21153 raft_consensus.cc:2379] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:13:44.640817 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 242796us
I20260812 19:13:44.641396 24238 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:44.641774 21153 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1619) took 2551 ms (client timeout 1604 ms). Trace:
W20260812 19:13:44.292203 24239 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.637s user 0.005s sys 0.010s
I20260812 19:13:44.642182 21150 raft_consensus.cc:1226] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
W20260812 19:13:44.642249 21153 rpcz_store.cc:269] 0812 19:13:42.090164 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.092187 (+ 2023us) service_pool.cc:227] Handling call
0812 19:13:44.641384 (+2549197us) spinlock_profiling.cc:243] Waited 2.54 s on lock 0x7b5400325854. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:44.641677 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5843734272}
I20260812 19:13:44.492483 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1582) took 1474 ms. Trace:
W20260812 19:13:44.494045 24294 log.cc:924] Time spent T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 1.444s user 0.005s sys 0.000s
I20260812 19:13:44.491585 24287 raft_consensus.cc:2726] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:44.506122 23751 consensus_queue.cc:260] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:44.643430 21243 rpcz_store.cc:276] 0812 19:13:43.018329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:43.098687 (+ 80358us) service_pool.cc:227] Handling call
0812 19:13:44.491870 (+1393183us) spinlock_profiling.cc:243] Waited 6.9 ms on lock 0x7b54002567d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:44.492203 (+ 333us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15866112}
I20260812 19:13:44.644637 23751 raft_consensus.cc:3037] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:44.645005 21150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1880) took 2191 ms. Trace:
I20260812 19:13:44.645603 21150 rpcz_store.cc:276] 0812 19:13:42.453277 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:43.397619 (+944342us) service_pool.cc:227] Handling call
0812 19:13:44.641722 (+1244103us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:44.642849 (+ 1127us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.644644 (+ 1795us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400325850. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:44.644908 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2717247232}
W20260812 19:13:44.420969 22008 leader_election.cc:334] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.150s (SENT)
W20260812 19:13:44.583861 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 135091us
I20260812 19:13:44.584539 21154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1887) took 2119 ms. Trace:
W20260812 19:13:44.597086 21241 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s user 0.021s sys 0.014s
W20260812 19:13:44.596015 21319 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.398s user 0.006s sys 0.006s
W20260812 19:13:44.604753 21115 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.010s sys 0.017s
I20260812 19:13:44.607666 22047 raft_consensus.cc:1261] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260812 19:13:44.614387 22052 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.122s user 0.003s sys 0.011s
W20260812 19:13:44.599268 21112 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.272s user 0.013s sys 0.024s
I20260812 19:13:44.517035 21239 rpcz_store.cc:276] 0812 19:13:43.276734 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:43.307264 (+ 30530us) service_pool.cc:227] Handling call
0812 19:13:43.307743 (+ 479us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:43.308026 (+ 283us) raft_consensus.cc:1520] Early marking committed up to index 6
0812 19:13:43.915403 (+607377us) log.cc:842] Serialized 14 byte log entry
0812 19:13:44.393746 (+478343us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:44.393865 (+ 119us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:44.394005 (+ 140us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.394135 (+ 130us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.394408 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":805886,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":66353,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":133,"thread_start_us":1002669,"threads_started":2,"wal-append.queue_time_us":387687}
I20260812 19:13:44.618918 21237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1595) took 1329 ms. Trace:
W20260812 19:13:44.628906 21236 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.599s user 0.000s sys 0.012s
I20260812 19:13:44.481426 24196 raft_consensus.cc:2726] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:44.637684 24350 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:44.642256 21114 leader_election.cc:398] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6.
W20260812 19:13:44.494865 22048 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40598 (request call id 661) took 2667 ms (client timeout 2669 ms). Trace:
I20260812 19:13:44.453928 23764 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:44.646436 21243 raft_consensus.cc:1226] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:44.628937 24338 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:44.647544 24344 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:44.648375 21156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1859) took 2380 ms. Trace:
W20260812 19:13:44.338405 24336 log.cc:924] Time spent T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.131s user 0.000s sys 0.001s
I20260812 19:13:44.690594 23764 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:44.648746 24088 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:44.635309 24053 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:44.658242 21235 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 272) took 1705 ms. Trace:
I20260812 19:13:44.658836 23897 raft_consensus.cc:729] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:44.663291 21203 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:44.624907 21158 rpcz_store.cc:276] 0812 19:13:42.444376 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.497618 (+ 53242us) service_pool.cc:227] Handling call
0812 19:13:42.498040 (+ 422us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:42.498308 (+ 268us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:42.929219 (+430911us) log.cc:842] Serialized 14 byte log entry
0812 19:13:44.454938 (+1525719us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:44.455079 (+ 141us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:44.455189 (+ 110us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.455317 (+ 128us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.455588 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11628,"prepare.queue_time_us":125924,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":71,"raft.queue_time_us":169259,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":233,"thread_start_us":1922380,"threads_started":2,"wal-append.queue_time_us":1385697}
I20260812 19:13:44.695238 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1598) took 1015 ms. Trace:
W20260812 19:13:44.665385 21152 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.008s sys 0.045s
I20260812 19:13:44.698714 21243 rpcz_store.cc:276] 0812 19:13:43.673846 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:44.645204 (+971358us) service_pool.cc:227] Handling call
0812 19:13:44.646015 (+ 811us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:44.688253 (+ 42238us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.688559 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:44.699643 24338 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:44.700198 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:44.701274 24088 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:44.686710 21156 rpcz_store.cc:276] 0812 19:13:42.268080 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.269692 (+ 1612us) service_pool.cc:227] Handling call
0812 19:13:42.331324 (+ 61632us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:42.373506 (+ 42182us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:42.373545 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:42.374247 (+ 702us) log.cc:842] Serialized 52 byte log entry
0812 19:13:42.663655 (+289408us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:42.663713 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:42.663821 (+ 108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:44.315528 (+1651707us) spinlock_profiling.cc:243] Waited 35.6 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:44.647666 (+332138us) raft_consensus.cc:1671] finished
0812 19:13:44.647703 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.648148 (+ 445us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3061,"raft.queue_time_us":6423,"raft.run_cpu_time_us":5279,"raft.run_wall_time_us":35192,"spinlock_wait_cycles":81769856,"thread_start_us":236304,"threads_started":1,"wal-append.queue_time_us":443283}
I20260812 19:13:44.705350 21156 raft_consensus.cc:1226] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
I20260812 19:13:44.706166 21156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1892) took 1850 ms. Trace:
I20260812 19:13:44.695222 21235 rpcz_store.cc:276] 0812 19:13:42.952680 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.979153 (+ 26473us) service_pool.cc:227] Handling call
0812 19:13:42.979588 (+ 435us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:43.020704 (+ 41116us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:43.020751 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:43.020811 (+ 60us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:43.020859 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.657586 (+1636727us) spinlock_profiling.cc:243] Waited 40.8 ms on lock 0x7b54000ffad4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:44.657665 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.657999 (+ 334us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":93938560}
I20260812 19:13:44.596159 22053 rpcz_store.cc:276] 0812 19:13:42.764074 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.765115 (+ 1041us) service_pool.cc:227] Handling call
0812 19:13:42.765641 (+ 526us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:42.791575 (+ 25934us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:42.791617 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:42.806510 (+ 14893us) log.cc:842] Serialized 52 byte log entry
0812 19:13:43.223251 (+416741us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:43.223315 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:43.223429 (+ 114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:44.424599 (+1201170us) raft_consensus.cc:1671] finished
0812 19:13:44.424641 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.425086 (+ 445us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24836,"raft.queue_time_us":33068,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":33358,"thread_start_us":397812,"threads_started":1,"wal-append.queue_time_us":578488}
I20260812 19:13:44.693661 22047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 468) took 1272 ms. Trace:
I20260812 19:13:44.690594 24350 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:44.706584 21156 rpcz_store.cc:276] 0812 19:13:42.855734 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:44.704375 (+1848641us) service_pool.cc:227] Handling call
0812 19:13:44.704980 (+ 605us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:44.705862 (+ 882us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.706079 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:44.308444 24306 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:44.710467 21150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 312) took 2063 ms. Trace:
I20260812 19:13:44.711081 23764 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:44.671826 21287 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:44.675310 21154 rpcz_store.cc:276] 0812 19:13:42.455887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:44.453420 (+1997533us) service_pool.cc:227] Handling call
0812 19:13:44.454105 (+ 685us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:44.575146 (+121041us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.575481 (+ 335us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:44.718632 21150 rpcz_store.cc:276] 0812 19:13:42.647127 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:44.684972 (+2037845us) service_pool.cc:227] Handling call
0812 19:13:44.709811 (+ 24839us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:44.710081 (+ 270us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:44.710118 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:44.710182 (+ 64us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:44.710211 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.710268 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.710386 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:44.720433 24344 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:44.722909 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 51453.2us
I20260812 19:13:44.723950 21150 raft_consensus.cc:1261] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260812 19:13:44.724099 21287 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:44.724557 24053 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:44.724907 21288 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.798s user 0.003s sys 0.024s
I20260812 19:13:44.684250 21114 leader_election.cc:401] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260812 19:13:44.715209 24350 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:44.680369 21237 rpcz_store.cc:276] 0812 19:13:43.284239 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:43.342973 (+ 58734us) service_pool.cc:227] Handling call
0812 19:13:43.343473 (+ 500us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:43.343994 (+ 521us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:43.892090 (+548096us) log.cc:842] Serialized 14 byte log entry
0812 19:13:44.389924 (+497834us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:44.391335 (+ 1411us) log.cc:842] Serialized 26 byte log entry
0812 19:13:44.391644 (+ 309us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:44.392295 (+ 651us) log.cc:842] Serialized 14 byte log entry
0812 19:13:44.392536 (+ 241us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.392705 (+ 169us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:44.608806 (+216101us) raft_consensus.cc:1671] finished
0812 19:13:44.608837 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.613813 (+ 4976us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16386,"prepare.queue_time_us":1000461,"prepare.run_cpu_time_us":140,"prepare.run_wall_time_us":3326,"raft.queue_time_us":137190,"raft.run_cpu_time_us":684,"raft.run_wall_time_us":15910,"thread_start_us":887324,"threads_started":2,"wal-append.queue_time_us":394406}
W20260812 19:13:44.710062 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 289434us
I20260812 19:13:44.712110 22047 rpcz_store.cc:276] 0812 19:13:43.421503 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:43.425175 (+ 3672us) service_pool.cc:227] Handling call
0812 19:13:44.498002 (+1072827us) spinlock_profiling.cc:243] Waited 34 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:44.515683 (+ 17681us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:44.677437 (+161754us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.689101 (+ 11664us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":78178176}
W20260812 19:13:44.726715 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 84823.7us
W20260812 19:13:44.678258 21115 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.654s user 0.010s sys 0.017s
I20260812 19:13:44.728013 21114 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:44.684481 21151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1889) took 2224 ms. Trace:
I20260812 19:13:44.702324 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:44.517829 21324 rpcz_store.cc:276] 0812 19:13:43.181881 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:43.318376 (+136495us) service_pool.cc:227] Handling call
0812 19:13:43.318795 (+ 419us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:43.319070 (+ 275us) raft_consensus.cc:1520] Early marking committed up to index 7
0812 19:13:43.348944 (+ 29874us) log.cc:842] Serialized 14 byte log entry
0812 19:13:44.409462 (+1060518us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:44.409641 (+ 179us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:44.409770 (+ 129us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.409938 (+ 168us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.410289 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":33114,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":242397,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":163,"thread_start_us":1029751,"threads_started":2,"wal-append.queue_time_us":909035}
W20260812 19:13:44.707903 24346 log.cc:924] Time spent T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.390s user 0.000s sys 0.003s
W20260812 19:13:44.685979 22048 rpcz_store.cc:269] 0812 19:13:41.826888 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:41.833455 (+ 6567us) service_pool.cc:227] Handling call
0812 19:13:44.494211 (+2660756us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b5400452254. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:44.494581 (+ 370us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1231826176}
I20260812 19:13:44.708153 23897 consensus_queue.cc:260] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:44.708962 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1887) took 1595 ms. Trace:
I20260812 19:13:44.730778 21243 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:44.709928 21235 raft_consensus.cc:1261] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260812 19:13:44.721136 21154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 314) took 1859 ms. Trace:
I20260812 19:13:44.734251 21154 rpcz_store.cc:276] 0812 19:13:42.862272 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:44.719770 (+1857498us) service_pool.cc:227] Handling call
0812 19:13:44.720353 (+ 583us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:44.720682 (+ 329us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:44.720713 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:44.720800 (+ 87us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:44.720843 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.720901 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.721061 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:44.517602 21322 rpcz_store.cc:276] 0812 19:13:43.030990 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:43.031193 (+ 203us) service_pool.cc:227] Handling call
0812 19:13:43.031663 (+ 470us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:43.032167 (+ 504us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:43.032206 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:43.048613 (+ 16407us) log.cc:842] Serialized 26 byte log entry
0812 19:13:43.500563 (+451950us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:43.500623 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:43.500748 (+ 125us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:44.406635 (+905887us) raft_consensus.cc:1671] finished
0812 19:13:44.406674 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.407025 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3223,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":23639,"thread_start_us":451495,"threads_started":1,"wal-append.queue_time_us":861480}
I20260812 19:13:44.667832 21244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 273) took 1714 ms. Trace:
I20260812 19:13:44.730172 21151 rpcz_store.cc:276] 0812 19:13:42.456638 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:44.672862 (+2216224us) service_pool.cc:227] Handling call
0812 19:13:44.673443 (+ 581us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:44.677423 (+ 3980us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:44.677451 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:44.680353 (+ 2902us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:44.680402 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.680471 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.680618 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:44.708650 21239 raft_consensus.cc:1261] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260812 19:13:44.685348 21112 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.362s user 0.014s sys 0.025s
W20260812 19:13:44.732386 21288 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.843s user 0.003s sys 0.025s
I20260812 19:13:44.732875 21150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 315) took 1596 ms. Trace:
W20260812 19:13:44.737474 21152 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.697s user 0.008s sys 0.046s
I20260812 19:13:44.738122 21150 rpcz_store.cc:276] 0812 19:13:43.133558 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:44.722819 (+1589261us) service_pool.cc:227] Handling call
0812 19:13:44.723357 (+ 538us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:44.724461 (+ 1104us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.730132 (+ 5671us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:44.735903 21244 rpcz_store.cc:276] 0812 19:13:42.953450 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.953678 (+ 228us) service_pool.cc:227] Handling call
0812 19:13:42.954162 (+ 484us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:43.103594 (+149432us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:43.103625 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:43.103663 (+ 38us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:43.103700 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.667278 (+1563578us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b5400303c54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:44.667368 (+ 90us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.667622 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":343143168}
I20260812 19:13:44.734381 21155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 299) took 3670 ms. Trace:
I20260812 19:13:44.735157 24287 raft_consensus.cc:2726] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:44.743034 23897 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:44.745658 24292 raft_consensus.cc:2726] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:44.741495 21155 rpcz_store.cc:276] 0812 19:13:41.059495 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:41.167710 (+108215us) service_pool.cc:227] Handling call
0812 19:13:41.168043 (+ 333us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:44.384904 (+3216861us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:44.385017 (+ 113us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:44.385126 (+ 109us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:44.385221 (+ 95us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.729381 (+344160us) spinlock_profiling.cc:243] Waited 3.22 s on lock 0x7b5400194bd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:44.729437 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.729671 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":68887,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":290,"spinlock_wait_cycles":7395141504}
I20260812 19:13:44.748924 21233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1597) took 1414 ms. Trace:
I20260812 19:13:44.749621 21156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 316) took 1544 ms. Trace:
I20260812 19:13:44.750177 21156 rpcz_store.cc:276] 0812 19:13:43.205439 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:44.735461 (+1530022us) service_pool.cc:227] Handling call
0812 19:13:44.748969 (+ 13508us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:44.749267 (+ 298us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:44.749296 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:44.749345 (+ 49us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:44.749378 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.749439 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.749559 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:44.731729 24338 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:44.752499 21238 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.001s sys 0.010s
W20260812 19:13:44.759382 24058 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 2.524s user 0.018s sys 0.018s
W20260812 19:13:44.759985 24058 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.553s user 0.018s sys 0.018s
I20260812 19:13:44.729372 24085 raft_consensus.cc:2726] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6.
W20260812 19:13:44.767501 21238 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.814s user 0.001s sys 0.010s
W20260812 19:13:44.760080 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 1.43194e+06us
I20260812 19:13:44.738756 23764 raft_consensus.cc:2726] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 19:13:44.772081 24313 log.cc:924] Time spent T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 1.657s user 0.005s sys 0.010s
W20260812 19:13:44.771962 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 48118us
I20260812 19:13:44.773808 21287 leader_election.cc:398] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 2. Current term is 4.
I20260812 19:13:44.776393 21112 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:44.779702 23764 raft_consensus.cc:2781] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:44.780232 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1608) took 2643 ms. Trace:
I20260812 19:13:44.780550 23764 raft_consensus.cc:686] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:44.780854 21320 rpcz_store.cc:276] 0812 19:13:42.136605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.138795 (+ 2190us) service_pool.cc:227] Handling call
0812 19:13:42.659651 (+520856us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:42.704176 (+ 44525us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.779643 (+2075467us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b5400354650. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:44.779989 (+ 346us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1196961664}
I20260812 19:13:44.783294 23764 consensus_queue.cc:237] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
W20260812 19:13:44.794625 24277 log.cc:924] Time spent T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 2.269s user 0.022s sys 0.023s
I20260812 19:13:44.800227 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:44.755647 21321 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.615s user 0.004s sys 0.006s
W20260812 19:13:44.803231 21321 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.799s user 0.004s sys 0.007s
I20260812 19:13:44.803560 21034 catalog_manager.cc:5495] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 2 to 5, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 5 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:13:44.805735 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1623) took 1623 ms. Trace:
I20260812 19:13:44.733718 21319 rpcz_store.cc:276] 0812 19:13:43.113259 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:43.197145 (+ 83886us) service_pool.cc:227] Handling call
0812 19:13:43.197499 (+ 354us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:43.197728 (+ 229us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:43.198099 (+ 371us) log.cc:842] Serialized 14 byte log entry
0812 19:13:44.708200 (+1510101us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:44.708332 (+ 132us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:44.708462 (+ 130us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.708600 (+ 138us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.708886 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":25377,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":2296,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":183,"thread_start_us":1477841,"threads_started":1,"wal-append.queue_time_us":1127515}
I20260812 19:13:44.813190 24301 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:44.813437 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:44.817339 24301 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:44.819211 22048 raft_consensus.cc:1226] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 6. Current term is 7. Ops: []
I20260812 19:13:44.806357 21321 rpcz_store.cc:276] 0812 19:13:43.182224 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:43.345518 (+163294us) service_pool.cc:227] Handling call
0812 19:13:43.398342 (+ 52824us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:43.975711 (+577369us) raft_consensus.cc:1520] Early marking committed up to index 2
0812 19:13:44.004270 (+ 28559us) log.cc:842] Serialized 14 byte log entry
0812 19:13:44.804108 (+799838us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:44.804277 (+ 169us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:44.804396 (+ 119us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.805318 (+ 922us) spinlock_profiling.cc:243] Waited 577 ms on lock 0x7b5400354b54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:44.805383 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.805648 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":28837,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":59,"raft.queue_time_us":1490,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":151,"spinlock_wait_cycles":1327058688,"thread_start_us":799172,"threads_started":1,"wal-append.queue_time_us":710389}
I20260812 19:13:44.749469 21233 rpcz_store.cc:276] 0812 19:13:43.334371 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:43.403846 (+ 69475us) service_pool.cc:227] Handling call
0812 19:13:43.487909 (+ 84063us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:43.547769 (+ 59860us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:43.547814 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:43.573795 (+ 25981us) log.cc:842] Serialized 26 byte log entry
0812 19:13:44.452358 (+878563us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:44.453055 (+ 697us) log.cc:842] Serialized 14 byte log entry
0812 19:13:44.620654 (+167599us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.623949 (+ 3295us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:44.707220 (+ 83271us) spinlock_profiling.cc:243] Waited 2.68 ms on lock 0x7b240011d8a8. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d69e2cf4 00007f52d69e0dfb 00007f52d6a6f559 00007f52d6a6f1e5 00007f52d6a6e82d 00007f52d971146e 00007f52d9728c0c 00007f52d9727f36 00007f52d9727b58 00007f52d9731551
0812 19:13:44.748420 (+ 41200us) raft_consensus.cc:1671] finished
0812 19:13:44.748560 (+ 140us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.748847 (+ 287us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2793,"prepare.queue_time_us":162996,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":4685,"raft.queue_time_us":86396,"raft.run_cpu_time_us":552,"raft.run_wall_time_us":65657,"spinlock_wait_cycles":6153600,"thread_start_us":877687,"threads_started":2,"wal-append.queue_time_us":585396}
I20260812 19:13:44.847504 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:44.850780 24129 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:44.851156 23855 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:44.851630 24129 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:44.806444 21034 catalog_manager.cc:5495] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 3 to 4, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 4 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:13:44.857100 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:44.856657 21034 catalog_manager.cc:5495] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 1 to 2, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 2 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:13:44.858774 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1869) took 2931 ms. Trace:
W20260812 19:13:44.859124 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 200807us
I20260812 19:13:44.859735 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:44.860751 24344 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:44.859233 21328 rpcz_store.cc:276] 0812 19:13:41.897923 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:41.898135 (+ 212us) service_pool.cc:227] Handling call
0812 19:13:41.913429 (+ 15294us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:41.913773 (+ 344us) raft_consensus.cc:1520] Early marking committed up to index 6
0812 19:13:42.061903 (+148130us) log.cc:842] Serialized 14 byte log entry
0812 19:13:42.222190 (+160287us) log.cc:842] Serialized 14 byte log entry
0812 19:13:42.643089 (+420899us) spinlock_profiling.cc:243] Waited 8.78 ms on lock 0x7b1800334de0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a764 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
0812 19:13:42.867229 (+224140us) spinlock_profiling.cc:243] Waited 29 ms on lock 0x7b1800334de0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a638 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
0812 19:13:44.145435 (+1278206us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b1800334de0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b6c345 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b87155 00007f52d9a9e3d0 00007f52d9ab0576 00007f52d9ac10d7 00007f52d9ac1058
0812 19:13:44.696742 (+551307us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:44.736278 (+ 39536us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:44.755288 (+ 19010us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.807857 (+ 52569us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.829251 (+ 21394us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54805,"prepare.queue_time_us":93756,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":76,"raft.queue_time_us":8413,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":18478,"spinlock_wait_cycles":115838720,"thread_start_us":828343,"threads_started":3,"wal-append.queue_time_us":362211,"wal-append.run_cpu_time_us":26631,"wal-append.run_wall_time_us":1463347}
I20260812 19:13:44.867048 23958 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:44.868201 21112 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:44.868405 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1589) took 1740 ms. Trace:
I20260812 19:13:44.851768 23855 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:44.887555 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 159754us
I20260812 19:13:44.854382 24129 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:44.892836 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 165241us
I20260812 19:13:44.870795 21238 rpcz_store.cc:276] 0812 19:13:43.128041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:43.266036 (+137995us) service_pool.cc:227] Handling call
0812 19:13:43.266389 (+ 353us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:43.283001 (+ 16612us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:43.283043 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:43.283616 (+ 573us) log.cc:842] Serialized 26 byte log entry
0812 19:13:43.911752 (+628136us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:44.821220 (+909468us) log.cc:842] Serialized 14 byte log entry
0812 19:13:44.867898 (+ 46678us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.868086 (+ 188us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:44.868157 (+ 71us) raft_consensus.cc:1671] finished
0812 19:13:44.868168 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.868338 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":299276,"prepare.run_cpu_time_us":126,"prepare.run_wall_time_us":4354,"raft.queue_time_us":24475,"raft.run_cpu_time_us":505,"raft.run_wall_time_us":26003,"thread_start_us":1406125,"threads_started":2,"wal-append.queue_time_us":808371}
I20260812 19:13:44.901775 21152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1876) took 2457 ms. Trace:
I20260812 19:13:44.850888 23432 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:44.912343 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 138898us
I20260812 19:13:44.890473 24238 raft_consensus.cc:2726] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:44.912287 24189 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:44.915246 24338 consensus_queue.cc:1035] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20260812 19:13:44.918850 24305 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:44.919967 24305 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:44.907254 23751 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:44.921542 23751 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:44.922788 24363 consensus_queue.cc:1046] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260812 19:13:44.924145 23751 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:44.924234 24269 raft_consensus.cc:2726] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:44.912353 24345 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:44.927394 24345 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:44.930719 23958 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:44.904168 21152 rpcz_store.cc:276] 0812 19:13:42.444691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.505628 (+ 60937us) service_pool.cc:227] Handling call
0812 19:13:42.528052 (+ 22424us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:42.528401 (+ 349us) raft_consensus.cc:1520] Early marking committed up to index 4
0812 19:13:43.359036 (+830635us) log.cc:842] Serialized 14 byte log entry
0812 19:13:44.822211 (+1463175us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:44.845468 (+ 23257us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:44.888940 (+ 43472us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.889055 (+ 115us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.901679 (+ 12624us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11846,"prepare.queue_time_us":123798,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":142535,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":8771,"thread_start_us":2115690,"threads_started":3,"wal-append.queue_time_us":924232}
I20260812 19:13:44.933712 23958 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:44.922009 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 275) took 1967 ms. Trace:
I20260812 19:13:44.943070 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:44.943456 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:44.943250 24189 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:44.941598 24344 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:44.940755 21240 rpcz_store.cc:276] 0812 19:13:42.954208 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.954345 (+ 137us) service_pool.cc:227] Handling call
0812 19:13:42.954781 (+ 436us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:43.100090 (+145309us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:43.100141 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:43.100202 (+ 61us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:43.100248 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.921531 (+1821283us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54001509d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:44.921601 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.921829 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":333519232}
I20260812 19:13:44.946376 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:44.955714 21203 leader_election.cc:302] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:44.958267 21203 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:44.963316 21287 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:44.964577 24325 raft_consensus.cc:2726] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:44.964839 24287 raft_consensus.cc:2726] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:44.963388 21034 catalog_manager.cc:5495] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 2 to 5, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 5 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:44.965715 21149 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:44.963384 24292 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:44.966461 23432 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:44.966717 24292 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:44.965960 21287 leader_election.cc:398] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6.
I20260812 19:13:44.969803 23432 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:44.958351 23855 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:44.970265 21287 leader_election.cc:401] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:13:44.972620 24287 raft_consensus.cc:2726] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6.
W20260812 19:13:44.973966 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 66444.7us
I20260812 19:13:44.969827 24292 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:44.974998 21114 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:44.946575 24344 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:44.979576 24345 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:44.981215 21241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1593) took 1717 ms. Trace:
I20260812 19:13:44.961301 22052 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 466) took 1602 ms. Trace:
I20260812 19:13:44.973695 21287 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:44.993160 21287 leader_election.cc:398] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5.
I20260812 19:13:44.993992 24292 raft_consensus.cc:2726] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:45.001850 24076 raft_consensus.cc:2726] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:44.975051 21034 catalog_manager.cc:5495] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 3 to 6, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 6 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:44.988744 21236 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 276) took 2034 ms. Trace:
I20260812 19:13:45.012552 23751 raft_consensus.cc:2726] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:45.013370 21157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 310) took 2504 ms. Trace:
W20260812 19:13:45.016806 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 68379.2us
W20260812 19:13:45.015295 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 147264us
I20260812 19:13:45.018453 21287 leader_election.cc:398] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6.
I20260812 19:13:45.029392 21287 leader_election.cc:401] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:13:45.028946 21157 rpcz_store.cc:276] 0812 19:13:42.508509 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:44.817346 (+2308837us) service_pool.cc:227] Handling call
0812 19:13:44.817958 (+ 612us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:44.818334 (+ 376us) raft_consensus.cc:1520] Early marking committed up to index 6
0812 19:13:44.819868 (+ 1534us) log.cc:842] Serialized 14 byte log entry
0812 19:13:44.907092 (+ 87224us) log.cc:842] Serialized 14 byte log entry
0812 19:13:44.997513 (+ 90421us) spinlock_profiling.cc:243] Waited 1.97 ms on lock 0x7b1800085180. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a638 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
0812 19:13:45.012688 (+ 15175us) spinlock_profiling.cc:243] Waited 1.04 ms on lock 0x7b1800085180. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b014ca 00007f52e01d6f99 00007f52d69c5990 00007f52d6b87864 00007f52d6b87155 00007f52d9a9e3d0 00007f52d9ab0576 00007f52d9ac10d7 00007f52d9ac1058
0812 19:13:45.012907 (+ 219us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:45.012971 (+ 64us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:45.013026 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:45.013113 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:45.013288 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":782,"prepare.queue_time_us":2009,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":112,"raft.queue_time_us":112990,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":228,"spinlock_wait_cycles":6918400,"thread_start_us":82310,"threads_started":1,"wal-append.queue_time_us":121883}
I20260812 19:13:45.014997 21034 catalog_manager.cc:5495] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:13:44.987000 22052 rpcz_store.cc:276] 0812 19:13:43.358935 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:43.359280 (+ 345us) service_pool.cc:227] Handling call
0812 19:13:43.420303 (+ 61023us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:43.420571 (+ 268us) raft_consensus.cc:1520] Early marking committed up to index 7
0812 19:13:44.693172 (+1272601us) log.cc:842] Serialized 14 byte log entry
0812 19:13:44.898079 (+204907us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:44.907200 (+ 9121us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:44.910251 (+ 3051us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.913080 (+ 2829us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.961234 (+ 48154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":130,"prepare.queue_time_us":266311,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":69,"raft.queue_time_us":8370,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":12336,"spinlock_wait_cycles":10237696,"thread_start_us":1341233,"threads_started":2,"wal-append.queue_time_us":192853}
I20260812 19:13:45.043671 21149 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1865) took 2603 ms. Trace:
I20260812 19:13:45.043735 22056 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 664) took 2204 ms. Trace:
I20260812 19:13:45.044725 21034 catalog_manager.cc:5495] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:45.046849 24109 raft_consensus.cc:2726] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:45.028230 21236 rpcz_store.cc:276] 0812 19:13:42.955922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.989686 (+ 33764us) service_pool.cc:227] Handling call
0812 19:13:42.990069 (+ 383us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:42.990307 (+ 238us) raft_consensus.cc:1520] Early marking committed up to index 6
0812 19:13:44.690439 (+1700132us) log.cc:842] Serialized 14 byte log entry
0812 19:13:44.902466 (+212027us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:44.909256 (+ 6790us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:44.941205 (+ 31949us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.961209 (+ 20004us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.988665 (+ 27456us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2517,"prepare.queue_time_us":1137417,"prepare.run_cpu_time_us":1212,"prepare.run_wall_time_us":4345,"raft.queue_time_us":10911,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":8947,"spinlock_wait_cycles":7737472,"thread_start_us":1840874,"threads_started":2,"wal-append.queue_time_us":216922}
I20260812 19:13:45.046015 24297 consensus_queue.cc:1046] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260812 19:13:45.044193 21149 rpcz_store.cc:276] 0812 19:13:42.440662 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.440857 (+ 195us) service_pool.cc:227] Handling call
0812 19:13:42.441248 (+ 391us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:44.923855 (+2482607us) spinlock_profiling.cc:243] Waited 302 ms on lock 0x7b54000afda0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9a3440d 00007f52d9a22050 00007f52d9a85aba 00007f52d9a9d3af 00007f52d9a9eb4e 00007f52d9aa8c89 00007f52d9aa9556 00007f52d9aa5edf 00007f52d9aa599e 00007f52de9e72e9
0812 19:13:45.043154 (+119299us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:45.043201 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:45.043275 (+ 74us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:45.043344 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:45.043464 (+ 120us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:45.043600 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44846,"raft.queue_time_us":46754,"spinlock_wait_cycles":695162752}
I20260812 19:13:45.016402 24338 raft_consensus.cc:3032] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:13:45.045428 22056 rpcz_store.cc:276] 0812 19:13:42.839362 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:42.839527 (+ 165us) service_pool.cc:227] Handling call
0812 19:13:42.839951 (+ 424us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:42.998267 (+158316us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:45.043193 (+2044926us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b5400622fd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:45.043485 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":361205120}
W20260812 19:13:45.034765 22008 leader_election.cc:334] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.550s (SENT)
I20260812 19:13:44.985848 21241 rpcz_store.cc:276] 0812 19:13:43.264322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:43.296108 (+ 31786us) service_pool.cc:227] Handling call
0812 19:13:43.296643 (+ 535us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:43.688449 (+391806us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:43.688499 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:43.708026 (+ 19527us) log.cc:842] Serialized 52 byte log entry
0812 19:13:44.689667 (+981641us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:44.711134 (+ 21467us) log.cc:842] Serialized 14 byte log entry
0812 19:13:44.766532 (+ 55398us) log.cc:842] Serialized 14 byte log entry
0812 19:13:44.766989 (+ 457us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:44.980542 (+213553us) spinlock_profiling.cc:243] Waited 384 ms on lock 0x7b5400304154. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:44.980613 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:44.980852 (+ 239us) raft_consensus.cc:1671] finished
0812 19:13:44.980874 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:44.981139 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36492,"prepare.queue_time_us":2468,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":78,"raft.queue_time_us":71601,"raft.run_cpu_time_us":479,"raft.run_wall_time_us":25575,"spinlock_wait_cycles":900400768,"thread_start_us":792935,"threads_started":1,"wal-append.queue_time_us":522907}
I20260812 19:13:45.018993 24188 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:44.988848 24344 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:45.049415 24189 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:45.015517 23897 raft_consensus.cc:2781] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election won for term 5
W20260812 19:13:45.054121 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 81619.7us
I20260812 19:13:45.060802 24305 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:45.060840 22008 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:45.056849 24269 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:45.058310 24338 raft_consensus.cc:729] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:45.081130 24375 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:45.066691 24292 raft_consensus.cc:2726] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6.
I20260812 19:13:45.080931 21153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 311) took 2563 ms. Trace:
I20260812 19:13:45.074194 23897 raft_consensus.cc:686] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:45.087114 21153 rpcz_store.cc:276] 0812 19:13:42.515142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:45.018487 (+2503345us) service_pool.cc:227] Handling call
0812 19:13:45.019079 (+ 592us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:45.019517 (+ 438us) raft_consensus.cc:1520] Early marking committed up to index 7
0812 19:13:45.020053 (+ 536us) log.cc:842] Serialized 14 byte log entry
0812 19:13:45.077667 (+ 57614us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:45.077728 (+ 61us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:45.077776 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:45.077856 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:45.078002 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":932,"prepare.queue_time_us":1494,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":87,"raft.queue_time_us":6333,"thread_start_us":52953,"threads_started":1}
I20260812 19:13:45.088644 23897 consensus_queue.cc:237] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:45.085868 24286 raft_consensus.cc:2726] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260812 19:13:45.092594 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 106004us
I20260812 19:13:45.092933 24189 consensus_queue.cc:1046] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
W20260812 19:13:45.093474 22008 leader_election.cc:334] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.632s (SENT)
I20260812 19:13:45.095552 23735 consensus_queue.cc:1046] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:13:45.095392 24338 consensus_queue.cc:260] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:45.096765 23735 raft_consensus.cc:3032] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:13:45.097250 23735 raft_consensus.cc:729] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:45.097692 24338 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:45.100212 21319 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260812 19:13:45.075639 24287 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:45.101970 24287 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:45.103726 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:45.104193 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:45.104497 24287 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:45.105136 21320 raft_consensus.cc:2396] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260812 19:13:45.105763 21149 raft_consensus.cc:1261] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 19:13:45.106827 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:45.107662 21158 raft_consensus.cc:1261] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260812 19:13:45.108294 24301 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:45.108620 24297 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:45.107839 21321 raft_consensus.cc:2396] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260812 19:13:45.108985 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 134119us
I20260812 19:13:45.110102 21114 leader_election.cc:398] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5.
I20260812 19:13:45.110507 21114 leader_election.cc:401] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260812 19:13:45.111498 24129 raft_consensus.cc:2726] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5.
I20260812 19:13:45.110276 24167 raft_consensus.cc:3032] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:45.112882 21114 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:45.113842 24129 raft_consensus.cc:2726] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:45.098392 23735 consensus_queue.cc:260] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:45.112644 24167 raft_consensus.cc:729] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:45.118695 23735 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
W20260812 19:13:45.086246 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 68028.7us
I20260812 19:13:45.123767 21287 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:45.124141 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:45.125124 21319 raft_consensus.cc:2454] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:45.126098 24292 raft_consensus.cc:2726] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:45.105412 21318 raft_consensus.cc:1261] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260812 19:13:45.127478 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:45.082720 24305 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:45.132882 24305 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:45.137401 21287 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:45.142167 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1880) took 2125 ms. Trace:
I20260812 19:13:45.144721 21318 rpcz_store.cc:276] 0812 19:13:43.006600 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:43.007641 (+ 1041us) service_pool.cc:227] Handling call
0812 19:13:45.031956 (+2024315us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b5800271898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:45.079012 (+ 47056us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:45.127234 (+ 48222us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:45.131394 (+ 4160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2986368}
I20260812 19:13:45.136039 24305 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:45.169231 24325 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:45.173755 24292 raft_consensus.cc:2781] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:45.174364 24292 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:45.174823 24292 raft_consensus.cc:3037] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:45.180526 21236 raft_consensus.cc:1261] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260812 19:13:45.180361 24079 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:45.183885 24347 raft_consensus.cc:1050] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:13:45.184362 24079 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:45.185719 23425 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:45.188215 24345 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:45.189033 24345 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:45.189568 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:45.125747 24269 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:45.191649 24345 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:45.192863 24269 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:45.193593 24076 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:45.194362 24076 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:45.196087 23958 consensus_queue.cc:1035] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:13:45.193753 23750 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:45.137300 21034 catalog_manager.cc:5495] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 2 to 3, leader changed from 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 3 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260812 19:13:45.212616 22008 leader_election.cc:334] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.839s (SENT)
I20260812 19:13:45.223922 22008 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:45.237898 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 112457us
I20260812 19:13:45.260262 21238 raft_consensus.cc:1261] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260812 19:13:45.260865 21152 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:13:45.261750 21152 raft_consensus.cc:595] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:13:45.261106 24079 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:45.263947 24124 raft_consensus.cc:2726] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:45.253289 23750 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:45.272294 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 157653us
I20260812 19:13:45.276532 21240 raft_consensus.cc:1261] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260812 19:13:45.096869 24375 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:45.109849 24297 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:45.287062 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:45.287320 21287 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:45.086690 24188 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:45.119652 24167 consensus_queue.cc:260] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:45.288746 24287 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:45.289538 21287 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:45.171218 24325 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:45.289556 24287 raft_consensus.cc:504] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:45.290464 24305 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:45.290870 24188 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:45.292300 24287 leader_election.cc:288] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:45.289637 24375 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:45.217535 24076 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:45.217895 23958 consensus_queue.cc:1035] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260812 19:13:45.280874 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:45.304034 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:45.304808 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:45.217010 24286 consensus_queue.cc:1035] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.061s
I20260812 19:13:45.296653 21034 catalog_manager.cc:5495] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 2 to 3, leader changed from 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 3 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:13:45.305755 21241 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:13:45.257962 24345 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:45.291873 24152 raft_consensus.cc:2726] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:45.325841 21114 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:45.333773 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:45.300585 24287 raft_consensus.cc:2781] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:45.291208 24305 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:45.338814 21149 raft_consensus.cc:1261] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260812 19:13:45.323158 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:45.333974 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 44604.4us
W20260812 19:13:45.345196 24384 log.cc:924] Time spent T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.303s user 0.005s sys 0.000s
I20260812 19:13:45.345451 21287 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:45.347514 21287 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:45.357846 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:45.362520 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 79086.9us
I20260812 19:13:45.320441 24325 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:45.381203 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:45.342069 24287 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:45.354377 24129 raft_consensus.cc:2726] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:45.399256 21321 raft_consensus.cc:1261] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260812 19:13:45.397938 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:45.405663 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:45.416044 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:45.360841 21034 catalog_manager.cc:5495] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 2 to 6, leader changed from 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 6 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:45.414968 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:45.382784 24152 raft_consensus.cc:3037] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:45.398357 24287 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:45.370486 24297 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:45.427259 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 79971us
I20260812 19:13:45.430171 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:45.412092 21204 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.70.132:39579 timed out after 1.688s (SENT) [suppressed 23 similar messages]
I20260812 19:13:45.436259 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:45.437206 21149 raft_consensus.cc:2454] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:45.439988 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:45.379106 24345 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:45.419030 23750 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:45.449872 21287 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:45.438570 24301 raft_consensus.cc:2781] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:45.456775 21324 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:45.463303 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:45.447607 24393 log.cc:924] Time spent T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260812 19:13:45.470569 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:45.443676 22046 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEARNER]: Advancing to term 7
W20260812 19:13:45.463164 24387 log.cc:924] Time spent T 41fc49d178354e629e97311498b381ec P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.264s user 0.001s sys 0.001s
I20260812 19:13:45.473687 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:45.503743 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:45.503896 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:45.504828 21329 raft_consensus.cc:2454] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:45.427048 24347 consensus_queue.cc:237] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 2, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:45.505908 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:45.507365 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:45.493665 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 99207.8us
I20260812 19:13:45.495450 24305 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:45.498730 24383 log.cc:924] Time spent T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.160s user 0.001s sys 0.000s
I20260812 19:13:45.500895 24088 raft_consensus.cc:979] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059: : Instructing follower 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:13:45.511333 21329 raft_consensus.cc:2396] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260812 19:13:45.476467 21034 catalog_manager.cc:5495] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 2 to 4, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 4 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:13:45.522840 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:45.523368 21112 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:45.525797 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:45.527064 21323 raft_consensus.cc:1261] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260812 19:13:45.528975 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:45.529846 21321 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:45.530872 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:45.531642 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:45.532056 22055 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:45.533689 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:45.542028 21320 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:45.476720 24301 raft_consensus.cc:686] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:45.556943 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:45.531046 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:45.551124 24129 raft_consensus.cc:2781] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:45.559069 24270 consensus_queue.cc:1035] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.013s
I20260812 19:13:45.565354 24129 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:45.565465 24088 raft_consensus.cc:1067] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Signalling peer 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:13:45.565891 24129 raft_consensus.cc:3037] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:45.563555 23432 consensus_queue.cc:1035] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.004s
I20260812 19:13:45.571066 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:45.571516 21114 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:45.572598 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:45.572844 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:45.573045 21114 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:45.573410 21320 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:45.573693 21329 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:45.576327 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:45.582158 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:45.583033 21156 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:13:45.584039 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:45.583544 21034 catalog_manager.cc:5495] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 3 to 6, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 6 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:45.584736 21156 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
W20260812 19:13:45.586833 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 49726.1us
I20260812 19:13:45.574465 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:45.587898 21320 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:45.586259 21034 catalog_manager.cc:5495] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 3 to 4, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 4 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:45.561246 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:45.590689 21324 raft_consensus.cc:2454] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:45.597136 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:45.596506 24375 raft_consensus.cc:2726] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:45.598872 21203 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:45.602603 21112 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:45.660423 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 62248.3us
W20260812 19:13:45.664292 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 459725us
I20260812 19:13:45.667730 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:45.676786 21114 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:45.682924 21150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:13:45.683307 21114 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:45.683972 21150 raft_consensus.cc:595] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Received request to transfer leadership to TS d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:45.684549 24298 raft_consensus.cc:2781] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:45.685040 24298 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:45.686560 24298 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:45.689365 24301 consensus_queue.cc:237] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:45.693713 24345 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:45.694478 24345 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:45.694514 21200 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:45.695736 24053 raft_consensus.cc:2726] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:45.684499 21150 raft_consensus.cc:603] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:13:45.700930 21320 raft_consensus.cc:2396] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260812 19:13:45.775151 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:45.785605 21200 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:45.780745 21034 catalog_manager.cc:5495] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 5 to 6, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 6 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
W20260812 19:13:45.671720 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 72940.4us
W20260812 19:13:45.794739 22008 leader_election.cc:334] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.779s (SENT)
I20260812 19:13:45.795260 22008 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:45.788699 22053 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 LEARNER]: Advancing to term 5
I20260812 19:13:45.799863 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:45.797343 24088 raft_consensus.cc:2726] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260812 19:13:45.805151 23425 consensus_meta.cc:287] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.619s user 0.016s sys 0.000s
I20260812 19:13:45.809721 21153 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260812 19:13:45.698213 24189 raft_consensus.cc:2781] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:45.808030 24129 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:45.810246 21153 pending_rounds.cc:76] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059: Aborting all ops after (but not including) 4
I20260812 19:13:45.831736 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:45.808108 24338 consensus_meta.cc:287] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.710s user 0.008s sys 0.008s
I20260812 19:13:45.826030 21153 pending_rounds.cc:97] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260812 19:13:45.841630 21034 catalog_manager.cc:5495] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 3 to 4, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 4 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:45.673578 24109 consensus_queue.cc:1035] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.153s
I20260812 19:13:45.684525 24281 consensus_queue.cc:1035] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.162s
I20260812 19:13:45.848799 22046 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260812 19:13:45.864538 22046 pending_rounds.cc:76] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Aborting all ops after (but not including) 4
I20260812 19:13:45.864886 22046 pending_rounds.cc:97] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260812 19:13:45.865358 22046 raft_consensus.cc:2866] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 LEARNER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260812 19:13:45.849050 24401 log.cc:924] Time spent T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.228s user 0.000s sys 0.001s
W20260812 19:13:45.849892 24292 consensus_meta.cc:287] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.675s user 0.011s sys 0.003s
I20260812 19:13:45.851790 21034 catalog_manager.cc:5495] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:13:45.857604 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:45.883810 21328 raft_consensus.cc:2454] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
W20260812 19:13:45.859133 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 75678.2us
I20260812 19:13:45.821857 24189 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:45.862802 24406 raft_consensus.cc:2726] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:45.809055 24345 raft_consensus.cc:2781] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:45.875507 21153 raft_consensus.cc:2866] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260812 19:13:45.880223 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 63988us
I20260812 19:13:45.871631 24129 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:45.828291 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:45.897706 21150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:13:45.888656 21203 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:45.923565 21203 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:45.889937 24189 raft_consensus.cc:3037] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:45.931259 23735 consensus_meta.cc:287] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.810s user 0.012s sys 0.003s
I20260812 19:13:45.893977 24345 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:45.901257 24325 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:45.943897 24053 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:45.949584 24053 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:45.952888 24053 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:45.894682 24292 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:45.907501 24188 consensus_queue.cc:1035] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.034s
I20260812 19:13:45.935058 23750 raft_consensus.cc:2781] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:13:45.935846 24345 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:45.907817 24287 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:45.983172 21319 raft_consensus.cc:1226] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: []
W20260812 19:13:45.974108 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 50691.1us
W20260812 19:13:45.972605 24152 consensus_meta.cc:287] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.541s user 0.013s sys 0.001s
I20260812 19:13:45.964509 23735 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:45.985246 21287 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.071s sys 0.106s
W20260812 19:13:45.994138 21287 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.535s user 0.071s sys 0.107s
I20260812 19:13:45.994515 21328 raft_consensus.cc:1226] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:45.995076 22053 raft_consensus.cc:1261] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEARNER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260812 19:13:45.971000 23750 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:45.973109 24188 consensus_queue.cc:1225] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer 93578836968649f6bfcaecc709871412 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260812 19:13:45.982977 24053 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:46.003832 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 180473us
I20260812 19:13:45.986056 24350 raft_consensus.cc:2781] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:46.007376 21034 catalog_manager.cc:5495] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 1 to 3, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 3 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:45.997829 24152 raft_consensus.cc:2726] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 19:13:46.003283 23750 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:13:45.972779 24325 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:46.072299 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 472782us
W20260812 19:13:46.071484 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 388284us
I20260812 19:13:46.072930 21322 raft_consensus.cc:2454] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:45.987278 24298 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:46.079641 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 642591us
I20260812 19:13:46.022595 24350 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:46.061934 24129 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:46.085347 22011 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:46.094309 21150 raft_consensus.cc:595] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:13:46.095111 21150 raft_consensus.cc:603] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:13:46.028618 21034 catalog_manager.cc:5495] T eadd5203f7ca4c04b02bd28bea6b591c P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 1 to 4, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 4 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260812 19:13:46.069654 24347 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.022s sys 0.031s
I20260812 19:13:46.020068 24338 consensus_queue.cc:1035] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
I20260812 19:13:46.098073 24341 raft_consensus.cc:2726] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:46.051963 24403 consensus_queue.cc:1035] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.063s
I20260812 19:13:46.036345 24053 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:46.021459 23735 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:46.115870 24292 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:46.075543 24270 consensus_queue.cc:1035] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260812 19:13:46.097620 21112 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
W20260812 19:13:46.108867 24347 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s user 0.022s sys 0.031s
I20260812 19:13:46.092681 24350 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:46.114339 24287 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:46.051864 24269 consensus_queue.cc:1035] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.085s
I20260812 19:13:46.147996 21112 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:46.137006 24403 consensus_queue.cc:1225] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer d1e1da0b9585460e8c365bf55679d4fa log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260812 19:13:46.136484 24298 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:46.138918 24167 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:46.139956 24406 raft_consensus.cc:2781] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:46.141173 24238 raft_consensus.cc:2781] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:46.166523 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:46.152865 21034 catalog_manager.cc:5495] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20260812 19:13:46.102208 24129 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:46.161336 24238 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:46.166642 24297 raft_consensus.cc:2781] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:46.190469 21034 catalog_manager.cc:5495] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 3 to 4, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 4 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } }
I20260812 19:13:46.198961 24365 raft_consensus.cc:2726] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:46.200659 24129 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:46.204553 21034 catalog_manager.cc:5495] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 2 to 6, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 6 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:46.216742 21034 catalog_manager.cc:5495] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } }
W20260812 19:13:46.227324 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 74269.9us
I20260812 19:13:46.229040 21114 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:46.232785 24325 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:46.239537 23735 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:46.245069 24341 raft_consensus.cc:2726] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260812 19:13:46.242522 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:46.247820 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:46.249123 21150 raft_consensus.cc:2379] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 7: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:46.250253 21157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e06f074e35674592bdc8360835fd2661"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:13:46.251185 21157 raft_consensus.cc:595] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:13:46.198382 24238 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:46.253468 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:46.225948 21034 catalog_manager.cc:5495] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 1 to 3, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 3 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:46.273520 21034 catalog_manager.cc:5495] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 3 to 4, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 4 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:46.275490 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:46.276468 21149 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:13:46.244910 24297 raft_consensus.cc:686] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:46.282636 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:46.283898 21158 raft_consensus.cc:2379] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 7: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:46.290378 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260812 19:13:46.292002 24406 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:46.292805 24406 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:46.293802 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260812 19:13:46.295781 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 65168.6us
I20260812 19:13:46.323079 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:46.319113 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:13:46.317564 21034 catalog_manager.cc:5495] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa reported cstate change: term changed from 1 to 4, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130). New cstate: current_term: 4 leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260812 19:13:46.339239 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:46.367244 24297 consensus_queue.cc:237] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:46.386251 21114 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:46.340554 24345 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:46.394585 24088 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:46.391075 24341 consensus_queue.cc:1035] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.013s
W20260812 19:13:46.434502 22008 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.027s sys 0.059s
W20260812 19:13:46.435009 22008 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.514s user 0.027s sys 0.059s
W20260812 19:13:46.435302 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 515128us
I20260812 19:13:46.403872 24088 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:46.402369 24053 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:46.447679 21320 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.044s sys 0.047s
W20260812 19:13:46.450232 21320 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s user 0.044s sys 0.047s
I20260812 19:13:46.407327 24129 consensus_queue.cc:1035] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.035s
W20260812 19:13:46.454743 21113 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.011s sys 0.067s
W20260812 19:13:46.459062 21113 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.728s user 0.012s sys 0.067s
I20260812 19:13:46.443480 24088 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:13:46.479811 22008 leader_election.cc:334] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.155s (SENT)
I20260812 19:13:46.489125 22008 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:46.527065 21200 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.000s sys 0.019s
W20260812 19:13:46.531965 21200 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.554s user 0.000s sys 0.019s
W20260812 19:13:46.534003 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 559735us
W20260812 19:13:46.529765 21203 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.003s sys 0.016s
I20260812 19:13:46.480686 24350 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:46.536806 21203 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.552s user 0.004s sys 0.016s
I20260812 19:13:46.479261 24406 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:46.519171 24269 raft_consensus.cc:2726] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260812 19:13:46.543445 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 558410us
W20260812 19:13:46.552448 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 93203.8us
W20260812 19:13:46.554275 21155 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.015s sys 0.007s
W20260812 19:13:46.556617 21151 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.034s sys 0.045s
W20260812 19:13:46.556972 21155 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.593s user 0.016s sys 0.007s
I20260812 19:13:46.553117 21200 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:46.557060 21151 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.537s user 0.035s sys 0.045s
I20260812 19:13:46.531544 24345 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:46.560411 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 475116us
I20260812 19:13:46.563172 21151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1906) took 1071 ms. Trace:
I20260812 19:13:46.565626 21200 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:46.568822 24345 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:46.569646 24345 consensus_queue.cc:1225] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer f741102bfe1441d1b453e5a838649059 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260812 19:13:46.574905 23735 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:46.575856 23735 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:46.578102 24281 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:46.551940 24422 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:46.559506 24301 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.013s sys 0.010s
W20260812 19:13:46.564072 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 410712us
I20260812 19:13:46.566959 21155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1908) took 1074 ms. Trace:
I20260812 19:13:46.593014 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:46.594077 21149 raft_consensus.cc:2379] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 6: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:46.567351 24053 raft_consensus.cc:2726] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:46.600773 22053 raft_consensus.cc:1226] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7. Ops: []
I20260812 19:13:46.565079 23432 raft_consensus.cc:2726] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:46.600397 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:46.605603 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:46.611224 21153 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
W20260812 19:13:46.587566 24189 consensus_meta.cc:287] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.629s user 0.014s sys 0.000s
W20260812 19:13:46.602973 22008 leader_election.cc:334] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.039s (SENT)
I20260812 19:13:46.609069 21154 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:13:46.613874 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:46.590667 24301 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s user 0.014s sys 0.010s
I20260812 19:13:46.626173 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:46.626329 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:46.588622 21155 rpcz_store.cc:276] 0812 19:13:45.492513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:45.518253 (+ 25740us) service_pool.cc:227] Handling call
0812 19:13:45.524389 (+ 6136us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:45.524703 (+ 314us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:13:45.959792 (+435089us) log.cc:842] Serialized 14 byte log entry
0812 19:13:46.562899 (+603107us) log.cc:842] Serialized 14 byte log entry
0812 19:13:46.566284 (+ 3385us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:46.566365 (+ 81us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:46.566434 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:46.566697 (+ 263us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:46.566874 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2939,"prepare.queue_time_us":10782,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":92,"raft.queue_time_us":8227,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":19519,"thread_start_us":1022232,"threads_started":2}
I20260812 19:13:46.627027 21329 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 5
I20260812 19:13:46.606891 21322 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f"
dest_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:51568
I20260812 19:13:46.627893 21322 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
W20260812 19:13:46.623327 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 344252us
I20260812 19:13:46.623126 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:46.621096 21203 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:46.628918 22052 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 7: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:46.630120 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:46.630869 21318 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:46.628336 21322 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:46.623207 21319 raft_consensus.cc:3037] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 6
I20260812 19:13:46.636348 22046 raft_consensus.cc:1261] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:13:46.634354 21149 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:46.639402 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:46.640646 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:46.648814 21203 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:46.655853 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:46.659822 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
W20260812 19:13:46.661185 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 76966.6us
I20260812 19:13:46.662415 24409 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:46.663116 24409 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:46.665611 24409 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:46.666581 24417 log.cc:924] Time spent T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
I20260812 19:13:46.623032 24427 raft_consensus.cc:2781] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:46.667594 24088 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:46.612720 23735 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:46.670125 23735 raft_consensus.cc:2781] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:46.674273 23735 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:46.674844 23735 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:46.668139 24427 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:46.686468 21149 raft_consensus.cc:1261] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260812 19:13:46.695398 23750 consensus_meta.cc:287] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.592s user 0.013s sys 0.001s
I20260812 19:13:46.606415 24281 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:46.696964 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:46.705420 21153 raft_consensus.cc:2379] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:46.652273 24347 raft_consensus.cc:2781] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:46.730818 24347 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:46.619980 24287 consensus_queue.cc:1046] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260812 19:13:46.688501 24427 raft_consensus.cc:3037] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:46.722954 24424 log.cc:924] Time spent T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
W20260812 19:13:46.724787 24423 log.cc:924] Time spent T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.193s user 0.001s sys 0.001s
I20260812 19:13:46.725455 24281 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:46.725564 22046 raft_consensus.cc:1261] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEARNER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260812 19:13:46.726464 24350 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:46.598099 24422 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:46.731348 21149 pending_rounds.cc:76] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059: Aborting all ops after (but not including) 3
I20260812 19:13:46.625556 24189 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:46.794081 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:46.807624 21149 pending_rounds.cc:97] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260812 19:13:46.809442 21149 raft_consensus.cc:2866] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:46.698580 23750 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:46.731351 24347 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:46.816495 21322 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:46.796921 24414 log.cc:924] Time spent T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.293s user 0.002s sys 0.000s
I20260812 19:13:46.795960 24053 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:46.847673 21322 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:46.806555 24432 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:46.798324 24409 raft_consensus.cc:3032] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:13:46.856078 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1920) took 1011 ms. Trace:
I20260812 19:13:46.805940 24088 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:46.857923 23750 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:46.718454 24238 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:46.875049 21329 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:46.738140 24167 consensus_queue.cc:1035] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.112s
W20260812 19:13:46.891234 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 236447us
W20260812 19:13:46.899080 22008 leader_election.cc:334] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.991s (SENT)
I20260812 19:13:46.899652 22008 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:46.876398 24281 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:46.878409 24350 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:46.876597 21320 rpcz_store.cc:276] 0812 19:13:45.845066 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:45.853947 (+ 8881us) service_pool.cc:227] Handling call
0812 19:13:45.854566 (+ 619us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:45.855230 (+ 664us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:45.855275 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:45.856038 (+ 763us) log.cc:842] Serialized 52 byte log entry
0812 19:13:46.451046 (+595008us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:46.451103 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:46.451214 (+ 111us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:46.855803 (+404589us) raft_consensus.cc:1671] finished
0812 19:13:46.855832 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:46.856003 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5491,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":162,"thread_start_us":594463,"threads_started":1,"wal-append.queue_time_us":773270}
I20260812 19:13:46.876427 24409 raft_consensus.cc:729] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:46.930173 21153 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:13:46.902652 24269 raft_consensus.cc:2726] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:46.956725 21200 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:46.877144 24053 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:46.897531 23425 consensus_queue.cc:1035] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.039s
I20260812 19:13:46.959304 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:46.890506 24432 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:46.985309 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:46.992724 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:46.993813 21153 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:13:46.919523 24350 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:46.997960 24350 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:46.919957 24281 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:46.996068 24301 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:47.000002 24301 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:47.007078 21233 raft_consensus.cc:1226] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260812 19:13:47.003000 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:47.013495 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 53495.1us
I20260812 19:13:47.020311 21319 raft_consensus.cc:2454] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:47.006742 21243 pending_rounds.cc:76] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa: Aborting all ops after (but not including) 4
I20260812 19:13:47.025391 21243 pending_rounds.cc:97] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20260812 19:13:47.025681 21243 raft_consensus.cc:2866] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:47.025799 21320 raft_consensus.cc:2396] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260812 19:13:47.030330 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:47.032626 21112 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:46.950325 23735 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:47.012907 24292 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:47.000996 24281 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:47.003784 24432 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:47.086422 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:47.087862 21320 raft_consensus.cc:2396] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260812 19:13:47.094326 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:47.097536 21319 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 7: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:47.059855 24350 raft_consensus.cc:2781] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:47.103850 21153 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 8.
I20260812 19:13:47.105191 21322 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:13:47.106997 21204 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:47.046532 24287 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:47.065555 24189 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:47.118906 21204 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:47.120373 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:47.137470 21204 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:47.139586 21204 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:47.141338 21204 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:47.143342 21204 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:47.103312 24350 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:47.155257 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:47.116153 24422 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:47.135924 24238 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:47.111002 24281 consensus_queue.cc:1046] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260812 19:13:47.143460 24301 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:47.056272 24409 consensus_queue.cc:260] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:47.155428 24350 raft_consensus.cc:3037] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:47.187808 24281 raft_consensus.cc:2726] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:47.185853 24434 raft_consensus.cc:2726] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:47.157734 22046 raft_consensus.cc:1226] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7. Ops: []
I20260812 19:13:47.186707 24461 raft_consensus.cc:2726] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:47.222790 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:47.223537 21156 raft_consensus.cc:3037] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:47.228199 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:47.229035 21321 raft_consensus.cc:2454] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:47.230822 23735 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:47.230588 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:47.145915 24446 consensus_queue.cc:1035] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.071s
I20260812 19:13:47.231518 21321 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
W20260812 19:13:47.232498 24416 log.cc:924] Time spent T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.729s user 0.003s sys 0.001s
I20260812 19:13:47.232839 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:47.180354 24437 log.cc:924] Time spent T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.158s user 0.001s sys 0.001s
I20260812 19:13:47.152345 24287 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:47.171252 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:47.243732 21153 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:47.182374 24439 log.cc:924] Time spent T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
I20260812 19:13:47.187558 24429 raft_consensus.cc:2726] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:47.202948 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:47.257020 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:47.209146 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 65951us
I20260812 19:13:47.182226 24301 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:47.214931 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:47.122503 24292 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:47.219380 24053 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:47.221333 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:47.230716 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:47.254256 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:47.273342 21324 raft_consensus.cc:2396] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260812 19:13:47.166543 24243 consensus_queue.cc:1046] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.183s
I20260812 19:13:47.244264 24409 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
W20260812 19:13:47.295084 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 262594us
I20260812 19:13:47.240278 23750 raft_consensus.cc:3032] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:47.297945 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:47.299095 21154 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
W20260812 19:13:47.286783 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 226230us
I20260812 19:13:47.337369 24459 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:47.338332 24459 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:47.360288 24347 consensus_meta.cc:287] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.513s user 0.001s sys 0.010s
I20260812 19:13:47.360976 24347 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:47.363961 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 62766.4us
I20260812 19:13:47.366989 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:47.367971 21319 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:47.281461 24281 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:47.318742 24297 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:47.319429 23750 raft_consensus.cc:729] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:47.434741 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
W20260812 19:13:47.438037 24412 log.cc:924] Time spent T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.939s user 0.001s sys 0.002s
I20260812 19:13:47.494721 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:47.496424 21152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1893) took 2226 ms. Trace:
I20260812 19:13:47.496803 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:47.288000 24301 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:47.440912 24281 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:46.563733 21151 rpcz_store.cc:276] 0812 19:13:45.491712 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:45.518090 (+ 26378us) service_pool.cc:227] Handling call
0812 19:13:45.521737 (+ 3647us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:45.522077 (+ 340us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:45.992917 (+470840us) log.cc:842] Serialized 14 byte log entry
0812 19:13:46.562707 (+569790us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:46.562766 (+ 59us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:46.562820 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:46.562955 (+ 135us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:46.563104 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4360,"prepare.queue_time_us":10942,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":6824,"thread_start_us":1006291,"threads_started":2}
I20260812 19:13:47.511021 24449 raft_consensus.cc:2726] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:47.430634 24432 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:48.263670 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50042 (request call id 322) took 1036 ms. Trace:
W20260812 19:13:47.940846 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 87524.4us
I20260812 19:13:47.992556 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:47.938498 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:48.266453 22055 raft_consensus.cc:2454] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:13:48.264571 21329 rpcz_store.cc:276] 0812 19:13:47.227225 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:47.227438 (+ 213us) service_pool.cc:227] Handling call
0812 19:13:47.227945 (+ 507us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:47.228458 (+ 513us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:47.228485 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:47.229101 (+ 616us) log.cc:842] Serialized 52 byte log entry
0812 19:13:47.654856 (+425755us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:47.804493 (+149637us) log.cc:842] Serialized 14 byte log entry
0812 19:13:47.804994 (+ 501us) log.cc:842] Serialized 14 byte log entry
0812 19:13:47.805182 (+ 188us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:47.805297 (+ 115us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:47.947457 (+142160us) spinlock_profiling.cc:243] Waited 34.9 ms on lock 0x7b1800316600. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b57f96 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a638 00007f52d6b8db11
0812 19:13:47.981413 (+ 33956us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b1800316600. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a638 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
0812 19:13:48.206812 (+225399us) spinlock_profiling.cc:243] Waited 75.3 ms on lock 0x7b1800316600. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a764 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
0812 19:13:48.263409 (+ 56597us) raft_consensus.cc:1671] finished
0812 19:13:48.263430 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:48.263596 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23336,"prepare.queue_time_us":161152,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":98,"raft.queue_time_us":629329,"raft.run_cpu_time_us":807,"raft.run_wall_time_us":134050,"spinlock_wait_cycles":539330048,"thread_start_us":122808,"threads_started":1,"wal-append.queue_time_us":360280}
I20260812 19:13:47.515405 21318 raft_consensus.cc:2454] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
W20260812 19:13:48.030812 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 944918us
W20260812 19:13:47.764897 21156 consensus_meta.cc:287] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.541s user 0.013s sys 0.000s
I20260812 19:13:47.560452 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:48.087150 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 790927us
I20260812 19:13:48.275718 21156 raft_consensus.cc:2454] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:47.561062 21154 raft_consensus.cc:3037] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:47.815034 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1646) took 1223 ms. Trace:
W20260812 19:13:47.931239 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 672385us
I20260812 19:13:47.551954 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:48.278476 21204 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:48.279090 21321 raft_consensus.cc:3037] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:47.644630 21320 raft_consensus.cc:2454] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:47.971751 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:48.281311 21327 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 7: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:47.993870 21322 pending_rounds.cc:76] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412: Aborting all ops after (but not including) 5
I20260812 19:13:48.281683 21322 pending_rounds.cc:97] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260812 19:13:48.281951 21322 raft_consensus.cc:2866] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:47.958034 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:48.276419 21156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1933) took 1226 ms. Trace:
I20260812 19:13:48.282941 21156 rpcz_store.cc:276] 0812 19:13:47.049753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:47.222452 (+172699us) service_pool.cc:227] Handling call
0812 19:13:48.276366 (+1053914us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:47.510355 24427 consensus_meta.cc:287] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.769s user 0.007s sys 0.003s
I20260812 19:13:47.727909 21153 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:13:48.277516 21323 rpcz_store.cc:276] 0812 19:13:46.591457 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:46.592001 (+ 544us) service_pool.cc:227] Handling call
0812 19:13:46.614288 (+ 22287us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:46.921873 (+307585us) raft_consensus.cc:1520] Early marking committed up to index 2
0812 19:13:47.282566 (+360693us) log.cc:842] Serialized 14 byte log entry
0812 19:13:47.407852 (+125286us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:47.408373 (+ 521us) log.cc:842] Serialized 26 byte log entry
0812 19:13:47.408510 (+ 137us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:47.408542 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:47.408611 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:47.814720 (+406109us) raft_consensus.cc:1671] finished
0812 19:13:47.814750 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:47.814942 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":241693,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":357312,"raft.run_cpu_time_us":325,"raft.run_wall_time_us":11692,"thread_start_us":674540,"threads_started":4,"wal-append.queue_time_us":253265}
I20260812 19:13:47.496964 21152 rpcz_store.cc:276] 0812 19:13:45.269704 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:45.282891 (+ 13187us) service_pool.cc:227] Handling call
0812 19:13:45.312672 (+ 29781us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:45.336807 (+ 24135us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:45.336858 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:45.579857 (+242999us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b2c0006b848. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d8495e11 00007f52d84a5442 00007f52d9ae1c11 00007f52d9aa658a 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772
0812 19:13:45.580446 (+ 589us) log.cc:842] Serialized 52 byte log entry
0812 19:13:45.989538 (+409092us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:46.022964 (+ 33426us) log.cc:842] Serialized 14 byte log entry
0812 19:13:46.086313 (+ 63349us) spinlock_profiling.cc:243] Waited 33.2 ms on lock 0x7b18002cde20. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a638 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
0812 19:13:46.093022 (+ 6709us) spinlock_profiling.cc:243] Waited 39.6 ms on lock 0x7b18002cde20. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b014ca 00007f52e01d6f99 00007f52d69c5990 00007f52d6b87864 00007f52d6b87155 00007f52dd47b167 00007f52d9ab0807 00007f52d9abff0f 00007f52d9abfe88
0812 19:13:46.093774 (+ 752us) log.cc:842] Serialized 14 byte log entry
0812 19:13:46.094146 (+ 372us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:46.096206 (+ 2060us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:46.391174 (+294968us) spinlock_profiling.cc:243] Waited 3.5 ms on lock 0x7b18002cde20. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a638 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
0812 19:13:46.402863 (+ 11689us) spinlock_profiling.cc:243] Waited 3.11 ms on lock 0x7b18002cde20. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a764 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
0812 19:13:46.499692 (+ 96829us) spinlock_profiling.cc:243] Waited 3.73 ms on lock 0x7b18002cde20. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a799 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
0812 19:13:47.188410 (+688718us) spinlock_profiling.cc:243] Waited 37.8 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:47.496146 (+307736us) raft_consensus.cc:1671] finished
0812 19:13:47.496176 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:47.496352 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50607,"prepare.queue_time_us":112404,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":121,"raft.queue_time_us":19935,"raft.run_cpu_time_us":581,"raft.run_wall_time_us":15913,"spinlock_wait_cycles":282490112,"thread_start_us":408482,"threads_started":1,"wal-append.queue_time_us":883984}
I20260812 19:13:48.090366 21150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1918) took 1505 ms. Trace:
I20260812 19:13:48.025779 21244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 289) took 1069 ms. Trace:
W20260812 19:13:48.203128 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 172477us
W20260812 19:13:48.257556 24409 consensus_meta.cc:287] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.939s user 0.010s sys 0.002s
I20260812 19:13:47.512419 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:48.276410 21155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1935) took 1204 ms. Trace:
I20260812 19:13:48.303014 21112 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:48.307111 21155 rpcz_store.cc:276] 0812 19:13:47.071835 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:47.498599 (+426764us) service_pool.cc:227] Handling call
0812 19:13:48.276333 (+777734us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:48.307534 21288 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
W20260812 19:13:48.155972 24468 log.cc:924] Time spent T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.421s user 0.005s sys 0.000s
W20260812 19:13:48.296547 21153 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1639) took 1275 ms (client timeout 1508 ms). Trace:
I20260812 19:13:48.298090 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 319) took 1136 ms. Trace:
I20260812 19:13:48.299144 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:47.728358 24475 log.cc:924] Time spent T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260812 19:13:48.029292 24088 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:48.111356 24487 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:47.639950 24053 raft_consensus.cc:3032] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Stepping down as leader of term 2
W20260812 19:13:47.894948 24350 consensus_meta.cc:287] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.642s user 0.008s sys 0.005s
I20260812 19:13:47.767885 24471 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:47.784857 24477 log.cc:924] Time spent T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.071s user 0.000s sys 0.001s
I20260812 19:13:47.803174 24188 raft_consensus.cc:2726] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:48.309435 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:48.312167 21153 rpcz_store.cc:269] 0812 19:13:47.021331 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:47.149092 (+127761us) service_pool.cc:227] Handling call
0812 19:13:48.296440 (+1147348us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:48.338485 21329 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:48.298736 21204 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:48.339439 21112 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:48.341092 21204 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:48.312935 21156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1635) took 1350 ms. Trace:
I20260812 19:13:47.561324 24462 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:48.309522 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:47.782001 24406 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:48.179656 24444 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:48.312408 21320 rpcz_store.cc:276] 0812 19:13:47.143908 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:47.144380 (+ 472us) service_pool.cc:227] Handling call
0812 19:13:48.280544 (+1136164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:47.506449 24292 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:48.299033 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 325) took 1070 ms. Trace:
I20260812 19:13:48.114069 24420 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:47.812389 24459 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:47.812721 24287 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:48.276290 24418 raft_consensus.cc:2741] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260812 19:13:48.306141 24427 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:48.319588 21203 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:47.516637 24301 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:47.610330 24482 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:48.325368 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:47.933411 24347 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:47.995491 22048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 489) took 1824 ms. Trace:
I20260812 19:13:48.028379 23425 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:48.272600 24433 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:48.326586 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:48.327431 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:48.328104 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:48.299119 21244 rpcz_store.cc:276] 0812 19:13:46.956174 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:46.956439 (+ 265us) service_pool.cc:227] Handling call
0812 19:13:46.970754 (+ 14315us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:47.264192 (+293438us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:47.264237 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:47.358279 (+ 94042us) log.cc:842] Serialized 26 byte log entry
0812 19:13:47.803784 (+445505us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:47.803857 (+ 73us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:47.803957 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:48.025477 (+221520us) raft_consensus.cc:1671] finished
0812 19:13:48.025504 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:48.025696 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":73976,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":31988,"spinlock_wait_cycles":1874176,"thread_start_us":549240,"threads_started":2,"wal-append.queue_time_us":616161}
I20260812 19:13:48.308562 21151 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:13:48.330305 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:48.333568 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:48.304364 23897 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:48.377136 23897 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:48.286329 24472 raft_consensus.cc:2726] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260812 19:13:48.378989 23425 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:48.379899 23897 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:48.374753 24462 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:48.384093 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:48.384622 21235 raft_consensus.cc:1226] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:48.385154 21241 raft_consensus.cc:2396] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260812 19:13:48.385937 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:48.374462 21323 raft_consensus.cc:2454] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:48.381670 23425 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:48.387040 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:48.387252 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:48.387562 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:48.298523 21150 rpcz_store.cc:276] 0812 19:13:46.585169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:46.585412 (+ 243us) service_pool.cc:227] Handling call
0812 19:13:46.601393 (+ 15981us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:46.630663 (+ 29270us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:46.630696 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:46.642479 (+ 11783us) log.cc:842] Serialized 52 byte log entry
0812 19:13:46.918253 (+275774us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:46.976615 (+ 58362us) log.cc:842] Serialized 14 byte log entry
0812 19:13:47.515836 (+539221us) spinlock_profiling.cc:243] Waited 4.28 ms on lock 0x7b180048eb60. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b014ca 00007f52e01d6f99 00007f52dd94ebcf 00007f52d6b8a7b6 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d
0812 19:13:47.516781 (+ 945us) log.cc:842] Serialized 14 byte log entry
0812 19:13:47.721482 (+204701us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b480012c258. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52de94d7f1 00007f52de9448e7 00007f52de94477a 00007f52dea3895f 00007f52dea38627 00007f52dea4cb2c 00007f52dea4cad8 00007f52dea4ca58 00007f52dea4ca13 00007f52dea4bcbc 00007f52dd9cdbe6
0812 19:13:47.727821 (+ 6339us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b180048eb60. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b57f96 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b57f96 00007f52d6b57f30
0812 19:13:47.728638 (+ 817us) spinlock_profiling.cc:243] Waited 4.35 ms on lock 0x7b180048eb60. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b57f96 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a638 00007f52d6b8db11
0812 19:13:47.829039 (+100401us) spinlock_profiling.cc:243] Waited 6.33 ms on lock 0x7b180048eb60. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a638 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
0812 19:13:48.027151 (+198112us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b180048eb60. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b014ca 00007f52e01d6f99 00007f52dd94ebcf 00007f52d6b8a7b6 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d
0812 19:13:48.089841 (+ 62690us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:48.089965 (+ 124us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:48.090027 (+ 62us) raft_consensus.cc:1671] finished
0812 19:13:48.090049 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:48.090272 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1575153,"prepare.queue_time_us":571612,"prepare.run_cpu_time_us":160,"prepare.run_wall_time_us":22978,"raft.queue_time_us":628443,"raft.run_cpu_time_us":3511,"raft.run_wall_time_us":1092144,"spinlock_wait_cycles":96683008,"thread_start_us":189402,"threads_started":1,"wal-append.queue_time_us":305400}
I20260812 19:13:48.343266 23432 raft_consensus.cc:2781] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:48.391076 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:48.391604 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:48.364859 24420 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:48.392856 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:48.394065 24420 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:48.371120 22048 rpcz_store.cc:276] 0812 19:13:46.161572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:46.172349 (+ 10777us) service_pool.cc:227] Handling call
0812 19:13:46.442951 (+270602us) spinlock_profiling.cc:243] Waited 46.1 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:46.458809 (+ 15858us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:46.515305 (+ 56496us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:46.519547 (+ 4242us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:46.605339 (+ 85792us) log.cc:842] Serialized 26 byte log entry
0812 19:13:46.918394 (+313055us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:46.921027 (+ 2633us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:46.948989 (+ 27962us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:47.930881 (+981892us) raft_consensus.cc:1671] finished
0812 19:13:47.933212 (+ 2331us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:47.978804 (+ 45592us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":16727,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":14430,"spinlock_wait_cycles":106041600,"thread_start_us":236717,"threads_started":1,"wal-append.queue_time_us":341650}
I20260812 19:13:48.383522 24427 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:48.335541 24350 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:48.344941 21327 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 8
I20260812 19:13:48.347436 21112 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:48.353431 21153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 325) took 1728 ms. Trace:
I20260812 19:13:48.329587 24448 raft_consensus.cc:2781] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:48.356372 21318 rpcz_store.cc:276] 0812 19:13:47.228561 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:47.230162 (+ 1601us) service_pool.cc:227] Handling call
0812 19:13:48.298867 (+1068705us) spinlock_profiling.cc:243] Waited 242 ms on lock 0x7b5400084b54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:48.298974 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":557582976}
I20260812 19:13:48.406810 24448 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:48.407285 24448 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:48.407387 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:48.408205 21318 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:48.409353 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:48.410053 21318 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:48.329986 24456 raft_consensus.cc:2781] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:48.411684 24456 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:48.412115 24456 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:48.410830 23425 raft_consensus.cc:2781] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:13:48.413173 23425 raft_consensus.cc:686] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:48.327292 24053 raft_consensus.cc:729] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:48.329809 24351 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:48.030297 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:48.301707 24432 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:47.606236 23750 consensus_queue.cc:260] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:48.368674 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 49170.9us
I20260812 19:13:48.431766 21203 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:48.433486 21203 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:48.350414 24467 raft_consensus.cc:2726] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:48.435580 24472 raft_consensus.cc:2781] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:13:48.436178 24472 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:48.436623 24472 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:13:48.437331 21329 raft_consensus.cc:2454] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:48.440078 24467 raft_consensus.cc:2781] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:48.440907 24467 raft_consensus.cc:686] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:48.441505 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:48.445756 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:48.342406 21156 rpcz_store.cc:276] 0812 19:13:46.962309 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:48.298830 (+1336521us) service_pool.cc:227] Handling call
0812 19:13:48.312861 (+ 14031us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:48.326650 24088 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:48.467942 24498 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:48.468837 24498 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:48.469544 24088 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:48.471392 24498 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:48.475023 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:48.477213 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:48.338255 24471 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:48.479206 24281 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:48.479568 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:48.374466 21152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1638) took 1384 ms. Trace:
I20260812 19:13:48.523375 21152 rpcz_store.cc:276] 0812 19:13:46.990142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:48.306097 (+1315955us) service_pool.cc:227] Handling call
0812 19:13:48.374400 (+ 68303us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:48.375833 21155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1637) took 1398 ms. Trace:
I20260812 19:13:48.524710 21155 rpcz_store.cc:276] 0812 19:13:46.981679 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:48.308166 (+1326487us) service_pool.cc:227] Handling call
0812 19:13:48.375765 (+ 67599us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:48.375404 21324 raft_consensus.cc:3037] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:48.376153 24347 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:48.386261 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:48.387758 21237 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:13:48.391201 23432 raft_consensus.cc:686] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:48.386804 21235 raft_consensus.cc:2354] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7.
I20260812 19:13:48.401298 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:47.498936 24297 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:48.404301 21153 rpcz_store.cc:276] 0812 19:13:46.625103 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:48.352824 (+1727721us) service_pool.cc:227] Handling call
0812 19:13:48.353084 (+ 260us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:48.353211 (+ 127us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:48.353228 (+ 17us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:48.353254 (+ 26us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:48.353273 (+ 19us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:48.353305 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:48.353379 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:48.406193 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 65032.5us
I20260812 19:13:48.366465 24461 raft_consensus.cc:2726] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:48.402284 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:48.369248 24406 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:48.420221 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 290) took 1460 ms. Trace:
I20260812 19:13:48.421681 21149 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1925) took 1801 ms. Trace:
I20260812 19:13:48.422420 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260812 19:13:48.408244 24470 log.cc:924] Time spent T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.227s user 0.001s sys 0.000s
I20260812 19:13:48.421072 24129 raft_consensus.cc:1050] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:13:48.423980 24465 raft_consensus.cc:2781] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election won for term 8
I20260812 19:13:48.424396 24458 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:48.332667 24487 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:48.424477 24053 consensus_queue.cc:260] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:48.425375 24351 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:48.426082 24432 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:48.426385 24350 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:48.427402 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:48.429231 22008 leader_election.cc:334] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.550s (SENT)
I20260812 19:13:48.368932 21319 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
W20260812 19:13:48.480432 24478 log.cc:924] Time spent T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.289s user 0.005s sys 0.000s
I20260812 19:13:48.480870 24492 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:48.480962 24471 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:48.480458 21327 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 8.
I20260812 19:13:48.404515 24482 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:48.482105 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:48.374881 21151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1634) took 1422 ms. Trace:
W20260812 19:13:48.502370 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 160301us
W20260812 19:13:48.502909 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 174276us
I20260812 19:13:48.503182 21321 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:48.534759 21156 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:48.488780 24462 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:48.366184 24444 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:48.414129 24433 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:48.442137 24467 consensus_queue.cc:237] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:48.990756 22011 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.689s user 0.028s sys 0.040s
W20260812 19:13:49.091382 22011 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.790s user 0.031s sys 0.042s
I20260812 19:13:49.091809 21151 rpcz_store.cc:276] 0812 19:13:46.952303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:47.512026 (+559723us) service_pool.cc:227] Handling call
0812 19:13:48.374820 (+862794us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:49.094432 21321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 329) took 1859 ms (client timeout 1550 ms). Trace:
I20260812 19:13:48.702350 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:49.095067 21321 rpcz_store.cc:269] 0812 19:13:47.234879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:47.254905 (+ 20026us) service_pool.cc:227] Handling call
0812 19:13:49.094346 (+1839441us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.096401 21318 raft_consensus.cc:2454] T 784d71ae37454b3a9e8023f268428468 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:48.676011 21329 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
W20260812 19:13:49.097956 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 668715us
I20260812 19:13:48.702260 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:49.100312 21237 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:13:48.623160 21152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 326) took 1995 ms. Trace:
I20260812 19:13:49.101074 21152 rpcz_store.cc:276] 0812 19:13:46.627606 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:48.524432 (+1896826us) service_pool.cc:227] Handling call
0812 19:13:48.622410 (+ 97978us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:48.622693 (+ 283us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:48.622741 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:48.622802 (+ 61us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:48.622841 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:48.622904 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:48.623072 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:48.623871 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:49.091058 21149 rpcz_store.cc:276] 0812 19:13:46.620320 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:46.692163 (+ 71843us) service_pool.cc:227] Handling call
0812 19:13:46.731112 (+ 38949us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:46.933410 (+202298us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:46.933441 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:46.934013 (+ 572us) log.cc:842] Serialized 52 byte log entry
0812 19:13:47.313924 (+379911us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:47.313981 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:47.314098 (+ 117us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:48.421440 (+1107342us) raft_consensus.cc:1671] finished
0812 19:13:48.421461 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:48.421621 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10519,"raft.queue_time_us":43214,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":101,"thread_start_us":477856,"threads_started":2,"wal-append.queue_time_us":611692}
I20260812 19:13:49.105366 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1648) took 2503 ms. Trace:
I20260812 19:13:48.976423 21324 raft_consensus.cc:2454] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:48.650063 21155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1926) took 1957 ms. Trace:
W20260812 19:13:49.098977 21154 consensus_meta.cc:287] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.803s user 0.009s sys 0.003s
I20260812 19:13:49.105813 21323 raft_consensus.cc:2379] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:49.101457 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:49.106869 21155 rpcz_store.cc:276] 0812 19:13:46.692569 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:48.525665 (+1833096us) service_pool.cc:227] Handling call
0812 19:13:48.649334 (+123669us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:48.649609 (+ 275us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:48.649645 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:48.649715 (+ 70us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:48.649763 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:48.649840 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:48.649995 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.112275 21149 raft_consensus.cc:1226] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 6. Current term is 7. Ops: []
I20260812 19:13:49.105819 21328 rpcz_store.cc:276] 0812 19:13:46.601767 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:46.601938 (+ 171us) service_pool.cc:227] Handling call
0812 19:13:46.603984 (+ 2046us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:46.631449 (+ 27465us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:46.896589 (+265140us) log.cc:842] Serialized 14 byte log entry
0812 19:13:47.803273 (+906684us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:48.278832 (+475559us) log.cc:842] Serialized 590 byte log entry
0812 19:13:48.281728 (+ 2896us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:48.281769 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:48.281838 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:49.105092 (+823254us) raft_consensus.cc:1671] finished
0812 19:13:49.105116 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:49.105272 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":518938,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":64,"raft.queue_time_us":287399,"raft.run_cpu_time_us":334,"raft.run_wall_time_us":8500,"thread_start_us":1092814,"threads_started":3,"wal-append.queue_time_us":1233919}
I20260812 19:13:49.113703 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:49.107160 21154 raft_consensus.cc:2454] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:13:49.114789 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:49.114979 21154 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1640) took 2067 ms (client timeout 2114 ms). Trace:
I20260812 19:13:49.115466 21327 raft_consensus.cc:2454] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
W20260812 19:13:49.115396 21154 rpcz_store.cc:269] 0812 19:13:47.047463 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:47.496478 (+449015us) service_pool.cc:227] Handling call
0812 19:13:49.114917 (+1618439us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.116474 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.117353 21329 raft_consensus.cc:2379] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 6: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:49.119143 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:49.119799 21321 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:49.120126 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:49.120793 21235 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:49.121718 21149 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 327) took 1009 ms. Trace:
I20260812 19:13:49.095472 21243 rpcz_store.cc:276] 0812 19:13:46.959534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:46.966734 (+ 7200us) service_pool.cc:227] Handling call
0812 19:13:47.006392 (+ 39658us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:47.317717 (+311325us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:47.317774 (+ 57us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:47.319096 (+ 1322us) log.cc:842] Serialized 52 byte log entry
0812 19:13:47.590927 (+271831us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:48.273174 (+682247us) log.cc:842] Serialized 14 byte log entry
0812 19:13:48.273492 (+ 318us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:48.273575 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:48.411668 (+138093us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b180043a680. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a638 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
0812 19:13:48.413763 (+ 2095us) raft_consensus.cc:1671] finished
0812 19:13:48.419206 (+ 5443us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:48.420137 (+ 931us) inbound_call.cc:173] Queueing success response
0812 19:13:48.810458 (+390321us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b180043a680. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a764 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
Metrics: {"mutex_wait_us":67252,"prepare.queue_time_us":190663,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":98321,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":19111,"spinlock_wait_cycles":72714880,"thread_start_us":710898,"threads_started":3,"wal-append.queue_time_us":690316}
I20260812 19:13:49.176476 21158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 323) took 2577 ms. Trace:
I20260812 19:13:48.521122 24341 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:48.501920 24448 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:49.177278 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 60712us
I20260812 19:13:49.199327 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:49.585171 21288 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:51552 dropped due to backpressure. The service queue is full; it has 50 items.
I20260812 19:13:49.124284 21238 raft_consensus.cc:1226] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:49.114790 21324 raft_consensus.cc:2362] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6. Re-sending same reply.
W20260812 19:13:49.178695 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 87022.5us
I20260812 19:13:49.647710 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 346) took 1175 ms. Trace:
I20260812 19:13:49.252439 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:49.184612 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 45326.5us
I20260812 19:13:49.122889 21154 raft_consensus.cc:1226] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:49.200155 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.649176 21154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 329) took 1535 ms. Trace:
W20260812 19:13:49.619145 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 356064us
I20260812 19:13:49.122251 21149 rpcz_store.cc:276] 0812 19:13:48.112699 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.111684 (+998985us) service_pool.cc:227] Handling call
0812 19:13:49.112059 (+ 375us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:49.121420 (+ 9361us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:49.121647 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.649976 21318 raft_consensus.cc:2396] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260812 19:13:49.649696 21154 rpcz_store.cc:276] 0812 19:13:48.113476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.116132 (+1002656us) service_pool.cc:227] Handling call
0812 19:13:49.116475 (+ 343us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:49.648972 (+532497us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:49.649128 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.574544 21235 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
W20260812 19:13:49.618458 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 172879us
I20260812 19:13:49.651121 21235 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1543) took 1165 ms. Trace:
I20260812 19:13:49.253453 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:49.194306 21157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1917) took 2609 ms. Trace:
W20260812 19:13:48.649142 24493 log.cc:924] Time spent T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
I20260812 19:13:49.652087 21244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1546) took 1165 ms. Trace:
I20260812 19:13:49.246012 21321 raft_consensus.cc:2454] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:49.193614 21152 raft_consensus.cc:1226] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:49.655859 21244 rpcz_store.cc:276] 0812 19:13:48.486460 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.177046 (+690586us) service_pool.cc:227] Handling call
0812 19:13:49.652013 (+474967us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.656411 21152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1931) took 2801 ms. Trace:
I20260812 19:13:49.656508 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1930) took 1476 ms. Trace:
I20260812 19:13:49.122022 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.178004 21158 rpcz_store.cc:276] 0812 19:13:46.599284 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:46.680740 (+ 81456us) service_pool.cc:227] Handling call
0812 19:13:46.681219 (+ 479us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:46.681685 (+ 466us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:13:46.949791 (+268106us) log.cc:842] Serialized 14 byte log entry
0812 19:13:48.026249 (+1076458us) spinlock_profiling.cc:243] Waited 41.2 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:48.089464 (+ 63215us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:48.090190 (+ 726us) log.cc:842] Serialized 52 byte log entry
0812 19:13:48.268576 (+178386us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:48.504577 (+236001us) log.cc:842] Serialized 14 byte log entry
0812 19:13:48.511144 (+ 6567us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:48.518927 (+ 7783us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:49.176197 (+657270us) raft_consensus.cc:1671] finished
0812 19:13:49.176225 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:49.176398 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":853837,"prepare.queue_time_us":1143529,"prepare.run_cpu_time_us":144,"prepare.run_wall_time_us":14973,"raft.queue_time_us":844686,"raft.run_cpu_time_us":500,"raft.run_wall_time_us":76606,"spinlock_wait_cycles":94705408,"thread_start_us":764472,"threads_started":3,"wal-append.queue_time_us":763568}
I20260812 19:13:49.656929 21321 rpcz_store.cc:276] 0812 19:13:48.180365 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.118844 (+938479us) service_pool.cc:227] Handling call
0812 19:13:49.656440 (+537596us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.657404 21328 raft_consensus.cc:2454] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:49.658366 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1937) took 1378 ms. Trace:
I20260812 19:13:49.581126 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.659865 21328 rpcz_store.cc:276] 0812 19:13:48.279897 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.121727 (+841830us) service_pool.cc:227] Handling call
0812 19:13:49.658301 (+536574us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.662230 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:49.648447 21323 raft_consensus.cc:2362] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5. Re-sending same reply.
I20260812 19:13:49.651616 21235 rpcz_store.cc:276] 0812 19:13:48.485245 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:48.731912 (+246667us) service_pool.cc:227] Handling call
0812 19:13:49.119964 (+388052us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b5800004598. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:13:49.650991 (+531027us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5682944}
I20260812 19:13:49.664484 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 333) took 1509 ms. Trace:
W20260812 19:13:49.122777 22008 leader_election.cc:334] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.883s (SENT)
I20260812 19:13:49.653491 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:49.665011 21318 rpcz_store.cc:276] 0812 19:13:48.155486 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.120830 (+965344us) service_pool.cc:227] Handling call
0812 19:13:49.664419 (+543589us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.665221 22008 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:49.648101 21324 rpcz_store.cc:276] 0812 19:13:48.472476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.113425 (+640949us) service_pool.cc:227] Handling call
0812 19:13:49.647529 (+534104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.199404 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.651921 21157 rpcz_store.cc:276] 0812 19:13:46.584682 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:46.584874 (+ 192us) service_pool.cc:227] Handling call
0812 19:13:46.614945 (+ 30071us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:46.845813 (+230868us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:46.845862 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:47.855958 (+1010096us) spinlock_profiling.cc:243] Waited 1.61 ms on lock 0x7b2c0006b848. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d8495e11 00007f52d84a5442 00007f52d9ae1c11 00007f52d9aa658a 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772
0812 19:13:47.873036 (+ 17078us) log.cc:842] Serialized 26 byte log entry
0812 19:13:48.271878 (+398842us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:48.488176 (+216298us) log.cc:842] Serialized 14 byte log entry
0812 19:13:49.090740 (+602564us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:49.193843 (+103103us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b5400214154. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:49.193926 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:49.193998 (+ 72us) raft_consensus.cc:1671] finished
0812 19:13:49.194016 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:49.194226 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11480,"prepare.queue_time_us":1099748,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":86,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":1696,"spinlock_wait_cycles":533798528,"thread_start_us":519171,"threads_started":2,"wal-append.queue_time_us":431905}
I20260812 19:13:49.647178 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:49.664009 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 334) took 1446 ms. Trace:
I20260812 19:13:49.666348 21319 raft_consensus.cc:2454] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:49.666764 21323 rpcz_store.cc:276] 0812 19:13:48.217971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.116832 (+898861us) service_pool.cc:227] Handling call
0812 19:13:49.663948 (+547116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.667002 21240 raft_consensus.cc:2379] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:49.667073 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 352) took 1192 ms. Trace:
I20260812 19:13:49.656826 21152 rpcz_store.cc:276] 0812 19:13:46.855219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.103499 (+2248280us) service_pool.cc:227] Handling call
0812 19:13:49.177435 (+ 73936us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:49.656084 (+478649us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:49.656352 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.532930 21156 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:49.199052 21241 raft_consensus.cc:2354] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5.
I20260812 19:13:49.665846 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.668162 21156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1940) took 1314 ms. Trace:
I20260812 19:13:49.668509 21321 raft_consensus.cc:2454] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:49.667523 21319 rpcz_store.cc:276] 0812 19:13:48.474577 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.114568 (+639991us) service_pool.cc:227] Handling call
0812 19:13:49.667010 (+552442us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.668696 21156 rpcz_store.cc:276] 0812 19:13:48.353916 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:48.479180 (+125264us) service_pool.cc:227] Handling call
0812 19:13:49.668100 (+1188920us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.668881 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:49.669771 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:48.792891 24292 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:49.343611 24468 log.cc:924] Time spent T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20260812 19:13:49.686492 21156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1937) took 1336 ms. Trace:
I20260812 19:13:49.667665 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.687530 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:49.466215 24470 log.cc:924] Time spent T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260812 19:13:49.404326 24350 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:49.669060 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 351) took 1194 ms. Trace:
I20260812 19:13:49.671196 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1541) took 1186 ms. Trace:
I20260812 19:13:48.583705 24347 consensus_queue.cc:1225] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer 63db10063a7d4eb2a9cc9dc952654706 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260812 19:13:49.671094 21200 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:49.672586 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:49.698141 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.698823 21243 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:13:49.698964 21324 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:48.731654 24287 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:48.847795 24456 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:49.675884 21241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1540) took 1192 ms. Trace:
I20260812 19:13:49.675719 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:48.847249 24498 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:49.678071 21327 raft_consensus.cc:2379] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 4: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:49.680166 24508 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:49.684664 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37894 (request call id 1537) took 1298 ms. Trace:
I20260812 19:13:49.663241 24418 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:49.708575 24508 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:49.708992 24418 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:49.660645 24341 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:49.712915 21238 rpcz_store.cc:276] 0812 19:13:48.386046 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:48.394040 (+ 7994us) service_pool.cc:227] Handling call
0812 19:13:48.420668 (+ 26628us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:49.652260 (+1231592us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:49.683323 (+ 31063us) spinlock_profiling.cc:243] Waited 692 ms on lock 0x7b54002c47d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:49.684337 (+ 1014us) spinlock_profiling.cc:243] Waited 26.3 ms on lock 0x7b54002c47d0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:49.684568 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1651951104}
W20260812 19:13:49.713629 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 398822us
W20260812 19:13:49.713762 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 590979us
I20260812 19:13:49.688594 21323 raft_consensus.cc:2396] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260812 19:13:49.648864 24053 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:49.716495 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1924) took 2350 ms. Trace:
W20260812 19:13:49.113898 24129 raft_consensus.cc:1054] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: Unable to promote non-voter 63db10063a7d4eb2a9cc9dc952654706: Illegal state: Leader has not yet committed an operation in its own term
I20260812 19:13:49.193562 24270 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:49.235740 24088 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:49.278537 24351 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:49.691195 21152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1936) took 1355 ms. Trace:
I20260812 19:13:49.727308 21152 rpcz_store.cc:276] 0812 19:13:48.336143 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.667829 (+1331686us) service_pool.cc:227] Handling call
0812 19:13:49.690122 (+ 22293us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:49.690416 (+ 294us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:49.690454 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:49.690519 (+ 65us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:49.690576 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:49.690942 (+ 366us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:49.691097 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.670939 21319 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 7
I20260812 19:13:49.692052 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:49.692966 24502 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:49.237229 24465 raft_consensus.cc:686] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 8 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:49.694844 21240 rpcz_store.cc:276] 0812 19:13:48.484435 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.249045 (+764610us) service_pool.cc:227] Handling call
0812 19:13:49.671132 (+422087us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.695254 21321 rpcz_store.cc:276] 0812 19:13:48.474210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.665557 (+1191347us) service_pool.cc:227] Handling call
0812 19:13:49.669003 (+ 3446us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.659461 24482 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:49.090725 23750 raft_consensus.cc:3037] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:49.700325 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.701259 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.697695 21237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1544) took 1206 ms. Trace:
I20260812 19:13:49.744122 21329 raft_consensus.cc:2454] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
I20260812 19:13:49.744300 21237 rpcz_store.cc:276] 0812 19:13:48.485669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.659294 (+1173625us) service_pool.cc:227] Handling call
0812 19:13:49.691645 (+ 32351us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.744679 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 347) took 1271 ms. Trace:
I20260812 19:13:49.695731 24444 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:49.702287 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:48.823240 23432 consensus_queue.cc:237] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:49.703279 24461 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:49.746842 21243 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:13:49.704308 24297 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:49.698184 24500 raft_consensus.cc:2726] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:49.757615 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:49.759030 21243 raft_consensus.cc:2396] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260812 19:13:49.760018 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:49.760699 21243 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:49.761925 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:49.706688 21200 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:49.706369 21241 rpcz_store.cc:276] 0812 19:13:48.483893 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:48.640075 (+156182us) service_pool.cc:227] Handling call
0812 19:13:49.675816 (+1035741us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:49.764389 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 57849.2us
I20260812 19:13:49.765262 21112 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:49.714565 22008 leader_election.cc:334] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 3.338s (SENT)
I20260812 19:13:48.899290 24472 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:49.717010 21322 rpcz_store.cc:276] 0812 19:13:47.366362 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:47.946189 (+579827us) service_pool.cc:227] Handling call
0812 19:13:47.980323 (+ 34134us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:48.335155 (+354832us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:48.335186 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:48.335624 (+ 438us) log.cc:842] Serialized 52 byte log entry
0812 19:13:48.521406 (+185782us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:48.521447 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:48.521519 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:49.716233 (+1194714us) raft_consensus.cc:1671] finished
0812 19:13:49.716248 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:49.716419 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5897,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":11773,"thread_start_us":157800,"threads_started":1,"wal-append.queue_time_us":1244962}
I20260812 19:13:49.689297 24433 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:49.199805 24167 consensus_queue.cc:1035] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.507s
I20260812 19:13:49.687050 24487 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:49.117204 24406 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:49.722342 21150 raft_consensus.cc:2379] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 7: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:49.722450 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:49.724084 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 291) took 2751 ms. Trace:
I20260812 19:13:49.726246 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:48.824156 23425 consensus_queue.cc:237] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:49.729393 21152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1941) took 1366 ms. Trace:
I20260812 19:13:49.113483 24492 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:49.113257 24458 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:49.742455 21236 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 285) took 2808 ms. Trace:
I20260812 19:13:49.745013 21329 rpcz_store.cc:276] 0812 19:13:48.472853 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.673305 (+1200452us) service_pool.cc:227] Handling call
0812 19:13:49.744606 (+ 71301us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.745520 21318 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:49.700948 24292 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:49.705278 24350 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:49.702939 24287 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:49.749377 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:49.750499 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:49.751380 24508 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:49.743589 24502 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:49.760594 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:49.735812 24088 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:49.766335 21200 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:49.773778 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:49.672144 21288 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:51552 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1953)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 344)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1670)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 356)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 354)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1940)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1942)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 343)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1952)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 350)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1954)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 355)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1671)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1927)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1928)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50042 (request call id 336)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50042 (request call id 337)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1932)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1939)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1941)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50042 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1656)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1657)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1658)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1659)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1660)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1663)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1665)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1666)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1667)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1668)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1673)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1674)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1675)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1943)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1944)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1945)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1946)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1947)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1949)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1950)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1676)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1677)
I20260812 19:13:49.763347 21155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 328) took 1650 ms. Trace:
I20260812 19:13:49.801440 21320 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
I20260812 19:13:49.777608 21152 rpcz_store.cc:276] 0812 19:13:48.363130 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.728415 (+1365285us) service_pool.cc:227] Handling call
0812 19:13:49.728790 (+ 375us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:49.729007 (+ 217us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:49.729036 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:49.729088 (+ 52us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:49.729135 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:49.729202 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:49.729325 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.779148 22008 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:49.778313 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:49.776981 21239 rpcz_store.cc:276] 0812 19:13:46.969448 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:46.969729 (+ 281us) service_pool.cc:227] Handling call
0812 19:13:47.004217 (+ 34488us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:47.063231 (+ 59014us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:47.063269 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:47.079035 (+ 15766us) log.cc:842] Serialized 52 byte log entry
0812 19:13:47.531562 (+452527us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:47.531622 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:47.531735 (+ 113us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:48.314063 (+782328us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:48.479920 (+165857us) spinlock_profiling.cc:243] Waited 8.26 ms on lock 0x7b240011d788. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d69e2cf4 00007f52d69e0dfb 00007f52d6a6f559 00007f52d6a6f1e5 00007f52d6a6e82d 00007f52d971146e 00007f52d9728c0c 00007f52d9727f36 00007f52d9727b58 00007f52d9731551
0812 19:13:49.720325 (+1240405us) raft_consensus.cc:1671] finished
0812 19:13:49.720341 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:49.720444 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12483,"raft.queue_time_us":67865,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":25038,"spinlock_wait_cycles":44572160,"thread_start_us":428877,"threads_started":2,"wal-append.queue_time_us":952648}
I20260812 19:13:49.784399 21112 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:49.743564 24418 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:49.718051 24498 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:49.788604 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.789492 21240 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:49.783228 21236 rpcz_store.cc:276] 0812 19:13:46.933650 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:46.948595 (+ 14945us) service_pool.cc:227] Handling call
0812 19:13:46.984489 (+ 35894us) raft_consensus.cc:1457] Updating replica for 3 ops
0812 19:13:47.035736 (+ 51247us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:47.035776 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0812 19:13:47.072517 (+ 36741us) log.cc:842] Serialized 78 byte log entry
0812 19:13:47.187172 (+114655us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b4800190158. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52de94d7f1 00007f52de9448e7 00007f52de94477a 00007f52dea3895f 00007f52dea38627 00007f52dea4cb2c 00007f52dea4cad8 00007f52dea4ca58 00007f52dea4ca13 00007f52dea4bcbc 00007f52dd9cdbe6
0812 19:13:47.254170 (+ 66998us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:47.771293 (+517123us) log.cc:842] Serialized 14 byte log entry
0812 19:13:47.771852 (+ 559us) log.cc:842] Serialized 14 byte log entry
0812 19:13:47.802204 (+ 30352us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:47.802340 (+ 136us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:48.407508 (+605168us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:49.742226 (+1334718us) raft_consensus.cc:1671] finished
0812 19:13:49.742246 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:49.742400 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23895,"prepare.queue_time_us":470534,"prepare.run_cpu_time_us":133,"prepare.run_wall_time_us":4027,"raft.queue_time_us":106192,"raft.run_cpu_time_us":3184,"raft.run_wall_time_us":122624,"spinlock_wait_cycles":51565184,"thread_start_us":342532,"threads_started":2,"wal-append.queue_time_us":915885}
I20260812 19:13:49.811368 21322 raft_consensus.cc:2454] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:49.687315 21156 rpcz_store.cc:276] 0812 19:13:48.336372 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.669750 (+1333378us) service_pool.cc:227] Handling call
0812 19:13:49.671867 (+ 2117us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:49.672079 (+ 212us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:49.672107 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:49.672152 (+ 45us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:49.672187 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:49.672263 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:49.672361 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.792829 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:49.800494 21155 rpcz_store.cc:276] 0812 19:13:48.113122 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.114059 (+1000937us) service_pool.cc:227] Handling call
0812 19:13:49.762640 (+648581us) spinlock_profiling.cc:243] Waited 5.23 ms on lock 0x7b5800001898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:49.762845 (+ 205us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:49.763031 (+ 186us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:49.763052 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:49.763099 (+ 47us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:49.763127 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:49.763179 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:49.763289 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12038400}
I20260812 19:13:49.806497 21238 raft_consensus.cc:1261] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260812 19:13:49.807693 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.816691 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37894 (request call id 1538) took 1430 ms. Trace:
I20260812 19:13:49.807916 21200 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:49.809140 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:49.808682 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 153349us
I20260812 19:13:49.818468 21238 rpcz_store.cc:276] 0812 19:13:48.386382 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.788081 (+1401699us) service_pool.cc:227] Handling call
0812 19:13:49.806037 (+ 17956us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:49.816214 (+ 10177us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:49.816606 (+ 392us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.821389 21328 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:49.729985 24270 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:49.816795 21318 raft_consensus.cc:2454] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:49.818010 21329 raft_consensus.cc:2454] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
I20260812 19:13:49.822345 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1940) took 1526 ms. Trace:
I20260812 19:13:49.823205 21318 rpcz_store.cc:276] 0812 19:13:48.295742 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.790913 (+1495171us) service_pool.cc:227] Handling call
0812 19:13:49.822270 (+ 31357us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.824249 24513 raft_consensus.cc:2781] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:49.824724 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:49.825498 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:49.826198 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:49.828329 24500 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260812 19:13:49.828598 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 114064us
I20260812 19:13:49.829106 24459 consensus_queue.cc:1046] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260812 19:13:49.857468 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:49.837357 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.886050 21318 raft_consensus.cc:3037] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:49.886708 21329 raft_consensus.cc:2396] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260812 19:13:49.808795 21319 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:13:49.888370 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.860770 21241 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:13:49.891357 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 343) took 1420 ms. Trace:
I20260812 19:13:49.858872 21236 raft_consensus.cc:2379] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 6: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:49.847424 24516 raft_consensus.cc:2726] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:13:49.862430 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.862926 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 342) took 1392 ms. Trace:
I20260812 19:13:49.918313 21153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1929) took 3175 ms. Trace:
I20260812 19:13:49.918429 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:49.919054 21153 rpcz_store.cc:276] 0812 19:13:46.742518 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.100799 (+2358281us) service_pool.cc:227] Handling call
0812 19:13:49.193261 (+ 92462us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:49.247712 (+ 54451us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:49.247758 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:49.247811 (+ 53us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:49.247857 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:49.918087 (+670230us) spinlock_profiling.cc:243] Waited 54.1 ms on lock 0x7b54003cf854. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:49.918137 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:49.918259 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":124524416}
W20260812 19:13:49.920548 21320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 349) took 1447 ms (client timeout 1550 ms). Trace:
W20260812 19:13:49.920979 21320 rpcz_store.cc:269] 0812 19:13:48.473532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:48.623424 (+149892us) service_pool.cc:227] Handling call
0812 19:13:49.920336 (+1296912us) spinlock_profiling.cc:243] Waited 706 ms on lock 0x7b54002e54d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:49.920476 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1623429376}
W20260812 19:13:49.862460 21150 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1942) took 1499 ms (client timeout 1834 ms). Trace:
I20260812 19:13:49.866930 24287 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:49.868582 24502 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:49.869953 21328 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:13:49.870136 24350 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:49.801797 24508 raft_consensus.cc:2781] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:49.864235 24292 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:49.873208 24448 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:49.874189 24456 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:49.919435 22055 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:13:49.875269 24498 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:49.874432 24420 raft_consensus.cc:2932] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Committing config change with OpId 3.4: config changed from index 2 to 4, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:13:49.859800 22052 raft_consensus.cc:2932] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:13:49.874003 24418 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:49.858274 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1942) took 1551 ms. Trace:
I20260812 19:13:49.892051 21329 rpcz_store.cc:276] 0812 19:13:48.471303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.836915 (+1365612us) service_pool.cc:227] Handling call
0812 19:13:49.891265 (+ 54350us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:49.922977 21150 rpcz_store.cc:269] 0812 19:13:48.363467 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:48.391327 (+ 27860us) service_pool.cc:227] Handling call
0812 19:13:49.862251 (+1470924us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54002cc754. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:49.862401 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3058430848}
I20260812 19:13:49.916481 21327 rpcz_store.cc:276] 0812 19:13:48.470887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.116132 (+645245us) service_pool.cc:227] Handling call
0812 19:13:49.862732 (+746600us) spinlock_profiling.cc:243] Waited 22.6 ms on lock 0x7b5400084b54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:49.862866 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":52069760}
I20260812 19:13:49.938846 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.939026 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.924625 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 344) took 1453 ms. Trace:
I20260812 19:13:49.939551 21324 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:49.919636 21319 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:49.939917 21323 rpcz_store.cc:276] 0812 19:13:48.471659 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.862125 (+1390466us) service_pool.cc:227] Handling call
0812 19:13:49.924554 (+ 62429us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.940215 21322 rpcz_store.cc:276] 0812 19:13:48.306869 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.812402 (+1505533us) service_pool.cc:227] Handling call
0812 19:13:49.858197 (+ 45795us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.912108 24444 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:49.933141 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:49.932121 21237 raft_consensus.cc:2379] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:49.936813 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:49.945928 21240 raft_consensus.cc:1203] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Deduplicated request from leader. Original: 2.3->[4.4-6.5] Dedup: 6.5->[]
I20260812 19:13:49.946445 21240 raft_consensus.cc:1226] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 6. Current term is 7. Ops: [4.4-6.5]
I20260812 19:13:49.937116 21288 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:13:49.939162 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.859457 24513 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:49.940927 21112 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:49.933429 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:49.948544 24513 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:49.859617 24494 consensus_queue.cc:1046] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.069s
I20260812 19:13:49.949292 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:49.950105 21319 raft_consensus.cc:2379] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 7: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:49.950387 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.951053 21324 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:49.951786 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.941995 21150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:13:49.952868 21150 raft_consensus.cc:595] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:13:49.945149 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.953704 21323 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:13:49.955154 21327 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 7: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:49.955578 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50042 (request call id 350) took 1481 ms. Trace:
I20260812 19:13:49.955814 21327 rpcz_store.cc:276] 0812 19:13:48.473869 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.938888 (+1465019us) service_pool.cc:227] Handling call
0812 19:13:49.955530 (+ 16642us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.937700 24472 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:49.959148 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.927915 24508 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:49.962251 24508 raft_consensus.cc:3037] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:49.963032 24492 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:49.931613 24347 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:49.951655 21112 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:49.952457 21328 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:49.961305 21329 raft_consensus.cc:2396] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260812 19:13:49.965813 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.965927 21328 raft_consensus.cc:1261] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:13:49.966827 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1927) took 2030 ms. Trace:
I20260812 19:13:49.967356 21328 rpcz_store.cc:276] 0812 19:13:47.935982 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.965224 (+2029242us) service_pool.cc:227] Handling call
0812 19:13:49.965578 (+ 354us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:49.966475 (+ 897us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:49.966750 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.971668 24297 raft_consensus.cc:2781] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:49.972329 24297 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:49.972725 24129 raft_consensus.cc:1050] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:13:49.972826 24297 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:49.973322 24129 raft_consensus.cc:1054] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: Unable to promote non-voter 63db10063a7d4eb2a9cc9dc952654706: Illegal state: Leader has not yet committed an operation in its own term
I20260812 19:13:49.974439 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:49.975322 21321 raft_consensus.cc:2379] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 6: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:49.975189 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:49.986802 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 1929) took 2050 ms. Trace:
I20260812 19:13:49.965833 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.968930 21320 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
I20260812 19:13:49.987681 21329 rpcz_store.cc:276] 0812 19:13:47.936625 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.938753 (+2002128us) service_pool.cc:227] Handling call
0812 19:13:49.986690 (+ 47937us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.966661 21323 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:49.996048 21328 raft_consensus.cc:1261] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260812 19:13:49.995805 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:49.996752 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50042 (request call id 336) took 1778 ms. Trace:
I20260812 19:13:49.997180 21328 rpcz_store.cc:276] 0812 19:13:48.221984 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.995287 (+1773303us) service_pool.cc:227] Handling call
0812 19:13:49.995675 (+ 388us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:49.996471 (+ 796us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:49.996692 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.999171 21112 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:49.999825 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1932) took 1733 ms. Trace:
I20260812 19:13:50.000845 21112 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:50.000365 21320 rpcz_store.cc:276] 0812 19:13:48.266792 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.998793 (+1732001us) service_pool.cc:227] Handling call
0812 19:13:49.999186 (+ 393us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:49.999414 (+ 228us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:49.999450 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:49.999504 (+ 54us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:49.999549 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:49.999622 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:49.999747 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.998555 21328 raft_consensus.cc:1226] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:50.002152 21112 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:49.996816 21324 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:13:50.013082 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1939) took 1717 ms. Trace:
I20260812 19:13:50.013574 21329 rpcz_store.cc:276] 0812 19:13:48.295545 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.012265 (+1716720us) service_pool.cc:227] Handling call
0812 19:13:50.012556 (+ 291us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:50.012760 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:50.012786 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:50.012825 (+ 39us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:50.012861 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:50.012923 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:50.013023 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:50.014803 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:50.016274 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:50.018642 21236 raft_consensus.cc:2379] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 6: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:49.831554 24465 consensus_queue.cc:237] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:50.021612 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:50.021729 21200 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:50.023025 21112 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:50.023093 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50042 (request call id 345) took 1532 ms. Trace:
I20260812 19:13:50.024885 21200 leader_election.cc:302] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:50.024585 21328 rpcz_store.cc:276] 0812 19:13:48.472023 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.997920 (+1525897us) service_pool.cc:227] Handling call
0812 19:13:49.998325 (+ 405us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:50.004520 (+ 6195us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:50.004790 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:49.944803 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:50.029322 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1941) took 1727 ms. Trace:
I20260812 19:13:49.944324 24406 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:50.034981 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:50.042021 21327 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:50.041462 21323 rpcz_store.cc:276] 0812 19:13:48.302553 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.015544 (+1712991us) service_pool.cc:227] Handling call
0812 19:13:50.028612 (+ 13068us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:50.028872 (+ 260us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:50.028907 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:50.028967 (+ 60us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:50.029013 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:50.029084 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:50.029238 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:50.052152 24517 log.cc:924] Time spent T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
I20260812 19:13:50.054177 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:50.055174 21328 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 8
I20260812 19:13:49.959627 24433 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:50.038516 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1928) took 2102 ms. Trace:
I20260812 19:13:49.963718 24456 raft_consensus.cc:2781] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:50.073158 21324 raft_consensus.cc:1261] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260812 19:13:50.035571 21200 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:50.067044 21323 raft_consensus.cc:1261] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260812 19:13:50.074412 21319 rpcz_store.cc:276] 0812 19:13:47.936309 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.966546 (+2030237us) service_pool.cc:227] Handling call
0812 19:13:50.037785 (+ 71239us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:50.038066 (+ 281us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:50.038102 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:50.038160 (+ 58us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:50.038214 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:50.038281 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:50.038435 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:50.077901 21321 raft_consensus.cc:1261] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260812 19:13:49.949340 24088 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:50.073971 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 349789us
I20260812 19:13:50.078823 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50042 (request call id 337) took 1815 ms. Trace:
I20260812 19:13:50.079437 21321 rpcz_store.cc:276] 0812 19:13:48.263097 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.011145 (+1748048us) service_pool.cc:227] Handling call
0812 19:13:50.077523 (+ 66378us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:50.078476 (+ 953us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:50.078750 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:50.011446 24292 raft_consensus.cc:2781] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:13:50.079435 21200 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:50.089824 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:50.090775 21150 raft_consensus.cc:2379] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 7: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:13:50.091015 22008 leader_election.cc:334] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.241s (SENT)
I20260812 19:13:50.091470 22008 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:50.093046 21320 raft_consensus.cc:1226] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:50.026943 24129 raft_consensus.cc:2781] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:13:50.100741 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:50.026626 24512 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:13:50.101454 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:49.954803 24444 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:50.124538 21318 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:50.128805 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:50.120689 21327 raft_consensus.cc:2362] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5. Re-sending same reply.
I20260812 19:13:50.117507 21324 raft_consensus.cc:1226] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:50.083199 24456 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:50.085376 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 255530us
I20260812 19:13:50.105047 24467 raft_consensus.cc:2781] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:13:50.106947 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:50.110224 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1659) took 1001 ms. Trace:
I20260812 19:13:50.111127 21319 raft_consensus.cc:1226] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:50.100268 24365 raft_consensus.cc:2781] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:50.114995 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:50.104152 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:49.945145 24418 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:50.036342 24492 raft_consensus.cc:3032] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:50.122315 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:50.126292 21323 raft_consensus.cc:1261] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260812 19:13:50.053560 24453 raft_consensus.cc:2781] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:50.141083 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1660) took 1032 ms. Trace:
I20260812 19:13:50.152655 21322 rpcz_store.cc:276] 0812 19:13:49.108548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.020340 (+911792us) service_pool.cc:227] Handling call
0812 19:13:50.119649 (+ 99309us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b5800271898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:50.119873 (+ 224us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:50.130506 (+ 10633us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:50.130544 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:50.140752 (+ 10208us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:50.140817 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:50.140885 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:50.141006 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23606016}
I20260812 19:13:50.144251 21328 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 8.
I20260812 19:13:50.149514 21203 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:50.143846 21329 rpcz_store.cc:276] 0812 19:13:49.108271 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.017342 (+909071us) service_pool.cc:227] Handling call
0812 19:13:50.109500 (+ 92158us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b5800271898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:50.109726 (+ 226us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:50.109940 (+ 214us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:50.109961 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:50.109996 (+ 35us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:50.110031 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:50.110086 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:50.110166 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30270208}
I20260812 19:13:50.145979 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:50.147272 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1666) took 1034 ms. Trace:
I20260812 19:13:50.157068 21319 rpcz_store.cc:276] 0812 19:13:49.112757 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.078224 (+965467us) service_pool.cc:227] Handling call
0812 19:13:50.110924 (+ 32700us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:50.146983 (+ 36059us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:50.147210 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:50.072559 24531 raft_consensus.cc:2781] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:50.161134 21150 raft_consensus.cc:2396] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260812 19:13:50.168120 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1665) took 1058 ms. Trace:
I20260812 19:13:50.168704 21323 rpcz_store.cc:276] 0812 19:13:49.109439 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.077798 (+968359us) service_pool.cc:227] Handling call
0812 19:13:50.078077 (+ 279us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:50.152198 (+ 74121us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:50.168014 (+ 15816us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:50.164278 21329 raft_consensus.cc:2932] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:13:50.151741 24492 raft_consensus.cc:729] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:50.130239 24502 raft_consensus.cc:2781] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:50.126147 24406 raft_consensus.cc:2726] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:50.120031 24512 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:50.140395 24456 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:50.136364 24298 raft_consensus.cc:2781] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:50.193336 21158 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:50.195255 24528 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:50.198945 22045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 512) took 1067 ms. Trace:
I20260812 19:13:50.096374 24433 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:50.199630 21238 raft_consensus.cc:1261] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260812 19:13:50.174021 24467 raft_consensus.cc:686] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:50.141567 24461 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:50.212342 21235 raft_consensus.cc:2454] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:13:50.169314 24453 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:50.151789 21157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 330) took 1847 ms. Trace:
I20260812 19:13:50.169130 24531 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:50.224368 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 201448us
I20260812 19:13:50.144907 24129 raft_consensus.cc:686] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:50.175668 24270 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:50.190600 24502 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:50.171604 24472 consensus_queue.cc:1035] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
W20260812 19:13:50.215410 24520 log.cc:924] Time spent T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.209s user 0.000s sys 0.006s
I20260812 19:13:50.244522 21112 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:50.244650 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 126329us
I20260812 19:13:50.242386 21157 rpcz_store.cc:276] 0812 19:13:48.304535 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.666679 (+1362144us) service_pool.cc:227] Handling call
0812 19:13:49.691189 (+ 24510us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:49.691391 (+ 202us) raft_consensus.cc:1520] Early marking committed up to index 7
0812 19:13:49.718978 (+ 27587us) log.cc:842] Serialized 14 byte log entry
0812 19:13:50.149274 (+430296us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:50.149575 (+ 301us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:50.150874 (+ 1299us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:50.151231 (+ 357us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:50.151715 (+ 484us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18280,"prepare.queue_time_us":28092,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":60194,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":15374,"thread_start_us":408575,"threads_started":1,"wal-append.queue_time_us":479854}
I20260812 19:13:50.247498 21154 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:13:50.214371 23750 raft_consensus.cc:2781] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election won for term 6
W20260812 19:13:50.254878 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 79056.3us
I20260812 19:13:50.206961 24512 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:13:50.211593 24298 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:50.241587 24434 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:50.246941 24453 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:50.346453 21149 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1938) took 1993 ms. Trace:
W20260812 19:13:50.363741 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 65137.1us
I20260812 19:13:50.363556 21149 rpcz_store.cc:276] 0812 19:13:48.353075 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.679771 (+1326696us) service_pool.cc:227] Handling call
0812 19:13:49.700337 (+ 20566us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:49.700707 (+ 370us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:49.700732 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:49.720130 (+ 19398us) log.cc:842] Serialized 564 byte log entry
0812 19:13:50.084331 (+364201us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:50.084366 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:50.084935 (+ 569us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:50.346034 (+261099us) raft_consensus.cc:1671] finished
0812 19:13:50.346070 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:50.346327 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":345067,"threads_started":1,"wal-append.queue_time_us":477307}
W20260812 19:13:50.358067 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 113658us
I20260812 19:13:50.357371 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:50.359112 21200 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:50.366093 21238 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:50.246577 24531 raft_consensus.cc:3037] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 2 FOLLOWER]: Advancing to term 3
W20260812 19:13:50.366570 22008 leader_election.cc:334] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.790s (SENT)
I20260812 19:13:50.367156 22008 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:50.320421 24492 consensus_queue.cc:260] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:50.368397 24492 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:50.313866 21150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:13:50.374651 21150 raft_consensus.cc:595] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Received request to transfer leadership to TS d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:50.379791 21150 raft_consensus.cc:603] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:13:50.362044 23750 raft_consensus.cc:686] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:13:50.341708 24053 consensus_meta.cc:287] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.615s user 0.012s sys 0.000s
I20260812 19:13:50.271886 24502 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
W20260812 19:13:50.351358 24530 log.cc:924] Time spent T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.094s user 0.001s sys 0.000s
I20260812 19:13:50.390866 21112 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:50.390091 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:50.392921 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:50.370975 23425 raft_consensus.cc:2781] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 19:13:50.362782 24298 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:50.239727 24461 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:50.385547 24498 raft_consensus.cc:3032] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:13:50.215775 22045 rpcz_store.cc:276] 0812 19:13:49.131484 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.176691 (+ 45207us) service_pool.cc:227] Handling call
0812 19:13:49.686140 (+509449us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:49.822487 (+136347us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:49.830675 (+ 8188us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:49.859633 (+ 28958us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:49.865420 (+ 5787us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:50.198650 (+333230us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b540049acd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:50.198703 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:50.198864 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":245135232}
I20260812 19:13:50.225864 24528 raft_consensus.cc:504] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:50.413683 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:50.415886 21154 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 8.
I20260812 19:13:50.422148 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:50.437105 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 62171.9us
I20260812 19:13:50.371243 24434 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:50.390053 24167 raft_consensus.cc:2726] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260812 19:13:50.451015 24351 raft_consensus.cc:2781] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:50.458599 21238 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:50.394532 24432 raft_consensus.cc:2726] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:50.460027 21151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1928) took 3766 ms. Trace:
I20260812 19:13:50.467855 21200 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:50.406456 24487 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:50.495127 24487 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:50.497562 24487 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:50.459045 24418 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:50.459569 24538 raft_consensus.cc:1050] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
W20260812 19:13:50.499007 24538 raft_consensus.cc:1054] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: Unable to promote non-voter 63db10063a7d4eb2a9cc9dc952654706: Illegal state: Leader has not yet committed an operation in its own term
I20260812 19:13:50.499338 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:50.459834 24297 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:50.500262 21238 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:50.501362 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:50.501890 24297 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:50.502810 24351 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:50.460587 23750 consensus_queue.cc:237] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:50.503345 24351 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:50.506482 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:50.482041 21151 rpcz_store.cc:276] 0812 19:13:46.693112 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.092804 (+2399692us) service_pool.cc:227] Handling call
0812 19:13:49.653081 (+560277us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:49.804585 (+151504us) raft_consensus.cc:1520] Early marking committed up to index 4
0812 19:13:49.833242 (+ 28657us) log.cc:842] Serialized 14 byte log entry
0812 19:13:50.137550 (+304308us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:50.137608 (+ 58us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:50.137667 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:50.200824 (+ 63157us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b18002503c0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b014ca 00007f52e01d6f99 00007f52dd94ebcf 00007f52d6b8a7b6 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d
0812 19:13:50.459774 (+258950us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b540019d7d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:50.459828 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:50.459960 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":102060,"prepare.queue_time_us":28936,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":57,"raft.queue_time_us":42803,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":11434,"spinlock_wait_cycles":374001920,"thread_start_us":261461,"threads_started":1,"wal-append.queue_time_us":353486}
W20260812 19:13:50.510962 24529 log.cc:924] Time spent T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
I20260812 19:13:50.511433 24528 leader_election.cc:288] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:50.512257 24433 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:50.412130 23425 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:50.531644 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 149757us
W20260812 19:13:50.537928 21243 consensus_meta.cc:287] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.713s user 0.011s sys 0.005s
I20260812 19:13:50.527817 23425 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:13:50.533090 24444 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:50.532851 24506 consensus_queue.cc:1035] T fc50dccc6345495aaec42b51dc30d299 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260812 19:13:50.379302 24129 consensus_queue.cc:237] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:50.533601 24527 log.cc:924] Time spent T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.270s user 0.001s sys 0.000s
I20260812 19:13:50.556555 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:50.557379 21151 raft_consensus.cc:2454] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 4.
I20260812 19:13:50.532011 21244 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:13:50.418682 24498 raft_consensus.cc:729] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:50.546255 21200 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:50.591034 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 44873.1us
I20260812 19:13:50.592154 21200 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706
W20260812 19:13:50.595074 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 204294us
I20260812 19:13:50.596468 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:50.528651 24472 raft_consensus.cc:2726] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:50.606930 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:50.610601 24506 consensus_queue.cc:1035] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 19:13:50.613781 24458 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:50.388159 24467 consensus_queue.cc:237] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:50.615559 24298 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:50.618129 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:50.619014 21153 raft_consensus.cc:3032] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:50.619364 21153 raft_consensus.cc:729] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:50.547907 21243 raft_consensus.cc:2454] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:50.548961 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:50.550503 24292 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:50.551265 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:50.633569 22008 leader_election.cc:334] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.187s (SENT)
I20260812 19:13:50.633898 21238 raft_consensus.cc:2379] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:50.620304 21153 consensus_queue.cc:260] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:50.634804 21153 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:50.636056 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:50.642012 21243 raft_consensus.cc:2379] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 6: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:50.643877 21200 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:50.644145 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:50.645252 21150 raft_consensus.cc:2379] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:50.660495 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:50.662657 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:50.663935 21243 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:50.664238 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:50.633867 24468 log.cc:924] Time spent T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260812 19:13:50.667218 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:50.530563 24531 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:50.669319 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37894 (request call id 1542) took 2184 ms. Trace:
I20260812 19:13:50.673022 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:50.672858 21239 rpcz_store.cc:276] 0812 19:13:48.484835 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.805808 (+1320973us) service_pool.cc:227] Handling call
0812 19:13:49.834966 (+ 29158us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:49.835304 (+ 338us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:49.835343 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:49.880085 (+ 44742us) log.cc:842] Serialized 26 byte log entry
0812 19:13:50.249271 (+369186us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:50.625163 (+375892us) log.cc:842] Serialized 14 byte log entry
0812 19:13:50.668859 (+ 43696us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:50.668974 (+ 115us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:50.669038 (+ 64us) raft_consensus.cc:1671] finished
0812 19:13:50.669055 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:50.669235 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":149682,"thread_start_us":558261,"threads_started":2,"wal-append.queue_time_us":473080}
I20260812 19:13:50.642928 24516 raft_consensus.cc:2781] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:50.674449 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:50.617740 24543 raft_consensus.cc:2781] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:50.717969 21236 raft_consensus.cc:1226] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 2. Current term is 3. Ops: []
I20260812 19:13:50.642483 24292 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:13:50.601109 24494 consensus_queue.cc:1046] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.067s
I20260812 19:13:50.631193 24524 raft_consensus.cc:2781] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:50.669826 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:50.703341 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:50.715878 21157 raft_consensus.cc:1226] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 6. Current term is 7. Ops: []
W20260812 19:13:50.722922 22011 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 1.628s (SENT) [suppressed 30 similar messages]
I20260812 19:13:50.642930 24458 consensus_queue.cc:1035] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
W20260812 19:13:50.723922 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 88415.1us
I20260812 19:13:50.727006 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:50.722165 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:50.604763 24512 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:50.684381 24298 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:50.665452 24538 consensus_queue.cc:1046] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
W20260812 19:13:50.720628 21287 connection.cc:232] RPC call timeout handler was delayed by -2.6510939220897853s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15080000000000002s for a response
W20260812 19:13:50.752182 21287 connection.cc:232] RPC call timeout handler was delayed by -2.6510939220897853s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15080000000000002s for a response
W20260812 19:13:50.752746 21287 connection.cc:232] RPC call timeout handler was delayed by -2.6429939221125096s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
W20260812 19:13:50.753150 21287 connection.cc:232] RPC call timeout handler was delayed by -2.6141939220251516s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1549s for a response
I20260812 19:13:50.753705 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:50.751725 21322 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.014s sys 0.016s
W20260812 19:13:50.754664 21322 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.599s user 0.014s sys 0.016s
W20260812 19:13:50.760008 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 114806us
I20260812 19:13:50.763182 21153 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
W20260812 19:13:50.753760 21287 connection.cc:232] RPC call timeout handler was delayed by -2.6141939220251516s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1549s for a response
W20260812 19:13:50.766781 21287 connection.cc:232] RPC call timeout handler was delayed by -2.6123939220560715s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
W20260812 19:13:50.769351 21287 connection.cc:232] RPC call timeout handler was delayed by -2.6123939220560715s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
W20260812 19:13:50.769834 21287 connection.cc:232] RPC call timeout handler was delayed by -2.5313939220504835s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16410000000000002s for a response
W20260812 19:13:50.770273 21287 connection.cc:232] RPC call timeout handler was delayed by -2.4845939220394939s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1693s for a response
W20260812 19:13:50.770581 21287 connection.cc:232] RPC call timeout handler was delayed by -2.4773939220467582s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17010000000000003s for a response
W20260812 19:13:50.770910 21287 connection.cc:232] RPC call timeout handler was delayed by -2.4773939220467582s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17010000000000003s for a response
W20260812 19:13:50.771224 21287 connection.cc:232] RPC call timeout handler was delayed by -2.4602939221076667s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17200000000000001s for a response
W20260812 19:13:50.771634 21287 connection.cc:232] RPC call timeout handler was delayed by -2.3792939221020788s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18100000000000002s for a response
W20260812 19:13:50.772017 21287 connection.cc:232] RPC call timeout handler was delayed by -2.2838939221110195s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1916s for a response
W20260812 19:13:50.772428 21287 connection.cc:232] RPC call timeout handler was delayed by -2.2838939221110195s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1916s for a response
I20260812 19:13:50.736456 24502 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:50.781049 24502 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:50.785339 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1945) took 1407 ms. Trace:
I20260812 19:13:50.788363 21243 raft_consensus.cc:1226] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 6. Current term is 7. Ops: []
W20260812 19:13:50.772804 21287 connection.cc:232] RPC call timeout handler was delayed by -2.2793939220719039s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19210000000000002s for a response
W20260812 19:13:50.791337 21287 connection.cc:232] RPC call timeout handler was delayed by -2.2793939220719039s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19210000000000002s for a response
W20260812 19:13:50.791577 22052 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.007s sys 0.028s
W20260812 19:13:50.791983 22052 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.572s user 0.007s sys 0.028s
W20260812 19:13:50.791814 21287 connection.cc:232] RPC call timeout handler was delayed by -2.2640939221018925s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1938s for a response
W20260812 19:13:50.792361 21287 connection.cc:232] RPC call timeout handler was delayed by -2.2640939221018925s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1938s for a response
W20260812 19:13:50.792757 21287 connection.cc:232] RPC call timeout handler was delayed by -2.2010939220199361s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2008s for a response
W20260812 19:13:50.793188 21287 connection.cc:232] RPC call timeout handler was delayed by -2.2010939220199361s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2008s for a response
W20260812 19:13:50.793586 21287 connection.cc:232] RPC call timeout handler was delayed by -2.1479939221171662s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20670000000000002s for a response
W20260812 19:13:50.793959 21287 connection.cc:232] RPC call timeout handler was delayed by -2.1056939220288768s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2114s for a response
W20260812 19:13:50.794329 21287 connection.cc:232] RPC call timeout handler was delayed by -2.1056939220288768s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2114s for a response
W20260812 19:13:50.794790 21287 connection.cc:232] RPC call timeout handler was delayed by -2.0777939220424742s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21450000000000002s for a response
W20260812 19:13:50.795182 21287 connection.cc:232] RPC call timeout handler was delayed by -2.0777939220424742s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21450000000000002s for a response
W20260812 19:13:50.795801 21287 connection.cc:232] RPC call timeout handler was delayed by -2.0156939220614731s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2214s for a response
W20260812 19:13:50.796203 21287 connection.cc:232] RPC call timeout handler was delayed by -2.0156939220614731s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2214s for a response
I20260812 19:13:50.787845 21327 rpcz_store.cc:276] 0812 19:13:49.378434 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.131994 (+753560us) service_pool.cc:227] Handling call
0812 19:13:50.171196 (+ 39202us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:50.171520 (+ 324us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:13:50.416873 (+245353us) log.cc:842] Serialized 14 byte log entry
0812 19:13:50.784268 (+367395us) log.cc:842] Serialized 14 byte log entry
0812 19:13:50.784580 (+ 312us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:50.784659 (+ 79us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:50.784805 (+ 146us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:50.785128 (+ 323us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:50.785264 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":560117,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"thread_start_us":554952,"threads_started":3}
W20260812 19:13:50.800776 21287 connection.cc:232] RPC call timeout handler was delayed by -2.0030939220450819s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22280000000000003s for a response
W20260812 19:13:50.801465 21287 connection.cc:232] RPC call timeout handler was delayed by -1.8734939220594242s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2372s for a response
W20260812 19:13:50.801865 21287 connection.cc:232] RPC call timeout handler was delayed by -1.8734939220594242s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2372s for a response
W20260812 19:13:50.802276 21287 connection.cc:232] RPC call timeout handler was delayed by -1.8428939220029861s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24060000000000004s for a response
I20260812 19:13:50.802891 21155 raft_consensus.cc:1226] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 7. Current term is 8. Ops: []
W20260812 19:13:50.802691 21287 connection.cc:232] RPC call timeout handler was delayed by -1.8428939220029861s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24060000000000004s for a response
W20260812 19:13:50.803787 21287 connection.cc:232] RPC call timeout handler was delayed by -1.614293922088109s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.266s for a response
W20260812 19:13:50.804188 21287 connection.cc:232] RPC call timeout handler was delayed by -1.614293922088109s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.266s for a response
W20260812 19:13:50.804534 21287 connection.cc:232] RPC call timeout handler was delayed by -1.5647939220070839s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2715s for a response
W20260812 19:13:50.805181 21287 connection.cc:232] RPC call timeout handler was delayed by -1.5305939220124856s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27530000000000004s for a response
W20260812 19:13:50.805879 21287 connection.cc:232] RPC call timeout handler was delayed by -1.5305939220124856s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27530000000000004s for a response
W20260812 19:13:50.790932 24281 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.022s sys 0.029s
I20260812 19:13:50.750638 24526 raft_consensus.cc:3032] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEADER]: Stepping down as leader of term 4
W20260812 19:13:50.806344 21287 connection.cc:232] RPC call timeout handler was delayed by -1.2965939220739529s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3013s for a response
W20260812 19:13:50.816184 21287 connection.cc:232] RPC call timeout handler was delayed by -1.2965939220739529s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3013s for a response
W20260812 19:13:50.817209 21287 connection.cc:232] RPC call timeout handler was delayed by -1.2101939220447093s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3109s for a response
I20260812 19:13:50.679359 24498 consensus_queue.cc:260] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:50.817665 21287 connection.cc:232] RPC call timeout handler was delayed by -1.1390939221018925s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31880000000000003s for a response
W20260812 19:13:50.818121 21287 connection.cc:232] RPC call timeout handler was delayed by -1.1390939221018925s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31880000000000003s for a response
W20260812 19:13:50.833760 24541 log.cc:924] Time spent T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
I20260812 19:13:50.816229 24526 raft_consensus.cc:729] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:50.834369 24536 consensus_queue.cc:1035] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.231s
W20260812 19:13:50.835633 24347 consensus_meta.cc:287] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.865s user 0.007s sys 0.007s
I20260812 19:13:50.836127 24347 raft_consensus.cc:2726] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:50.835052 24526 consensus_queue.cc:260] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:50.837680 24526 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:50.839912 24540 raft_consensus.cc:2781] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election won for term 8
I20260812 19:13:50.840647 24540 raft_consensus.cc:686] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [term 8 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:50.841598 24540 consensus_queue.cc:237] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:50.843258 21287 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:50.843508 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 105176us
W20260812 19:13:50.810880 24281 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.537s user 0.022s sys 0.030s
I20260812 19:13:50.848654 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:50.869968 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:50.870726 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:50.876883 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:50.882266 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:50.915753 21238 raft_consensus.cc:1261] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260812 19:13:50.818569 24461 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:50.862103 24513 consensus_meta.cc:287] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.906s user 0.020s sys 0.000s
I20260812 19:13:50.948235 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:50.919158 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:50.920794 21320 raft_consensus.cc:1261] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260812 19:13:50.807891 24351 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:50.952885 24559 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:50.953685 24559 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:50.955065 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:50.955693 21238 raft_consensus.cc:2362] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6. Re-sending same reply.
I20260812 19:13:50.956228 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:50.956709 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:50.956897 21318 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:50.957931 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:50.957872 21150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49361681c5c1421594d10cd52061ec44"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:13:50.958747 21150 raft_consensus.cc:595] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:13:50.959129 21150 raft_consensus.cc:603] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:13:50.959614 21327 raft_consensus.cc:1261] T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260812 19:13:50.962808 21324 raft_consensus.cc:2379] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 8: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
W20260812 19:13:50.976678 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 188883us
I20260812 19:13:50.956933 24559 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:50.949472 24492 consensus_meta.cc:287] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.580s user 0.006s sys 0.010s
I20260812 19:13:50.954377 24351 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:51.000778 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 157621us
I20260812 19:13:50.949668 24502 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:50.879366 24453 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:50.949985 24433 consensus_queue.cc:1035] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.079s
I20260812 19:13:51.022137 21287 leader_election.cc:398] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7.
I20260812 19:13:51.028127 21287 leader_election.cc:401] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:13:50.997064 24524 raft_consensus.cc:686] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:51.010922 24531 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:51.041239 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1684) took 1155 ms. Trace:
W20260812 19:13:51.049052 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 278133us
I20260812 19:13:50.959951 24513 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:51.063267 21153 raft_consensus.cc:1226] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
I20260812 19:13:51.066630 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1677) took 1390 ms. Trace:
I20260812 19:13:51.070403 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:51.067217 21322 rpcz_store.cc:276] 0812 19:13:49.675821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.154596 (+478775us) service_pool.cc:227] Handling call
0812 19:13:50.154894 (+ 298us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:50.155113 (+ 219us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:50.783195 (+628082us) log.cc:842] Serialized 14 byte log entry
0812 19:13:51.065909 (+282714us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:51.066191 (+ 282us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:51.066296 (+ 105us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:51.066378 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:51.066525 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44299,"prepare.queue_time_us":524600,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"thread_start_us":829235,"threads_started":2}
I20260812 19:13:51.071631 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:51.070130 21329 rpcz_store.cc:276] 0812 19:13:49.885646 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.155798 (+270152us) service_pool.cc:227] Handling call
0812 19:13:50.163748 (+ 7950us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:50.164089 (+ 341us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:13:50.271243 (+107154us) log.cc:842] Serialized 14 byte log entry
0812 19:13:50.751184 (+479941us) log.cc:842] Serialized 14 byte log entry
0812 19:13:50.975207 (+224023us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:50.975278 (+ 71us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:50.975334 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:51.040916 (+ 65582us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:51.041147 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25937,"prepare.queue_time_us":47025,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":5519,"raft.queue_time_us":468653,"raft.run_cpu_time_us":312,"raft.run_wall_time_us":25274,"spinlock_wait_cycles":9579904,"thread_start_us":694343,"threads_started":3,"wal-append.queue_time_us":392002}
I20260812 19:13:51.075485 21238 raft_consensus.cc:2396] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260812 19:13:51.073745 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:51.076337 21329 raft_consensus.cc:3032] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:51.076890 21329 raft_consensus.cc:729] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
W20260812 19:13:51.078850 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 56867.6us
I20260812 19:13:51.079654 21287 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:51.083539 24508 consensus_meta.cc:287] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.121s user 0.014s sys 0.000s
I20260812 19:13:51.083647 24292 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:51.084285 24508 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:51.085457 21239 raft_consensus.cc:1226] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:51.086215 24292 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:51.089891 24456 consensus_meta.cc:287] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.871s user 0.018s sys 0.000s
I20260812 19:13:51.091639 21287 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:51.057556 24559 raft_consensus.cc:2781] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:51.077847 21329 consensus_queue.cc:260] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:51.020344 24502 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:51.098989 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:51.020496 23425 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:51.114900 21329 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:51.126564 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:51.134143 21156 raft_consensus.cc:1226] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 8. Current term is 9. Ops: []
I20260812 19:13:51.091572 24512 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:51.143393 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 51864.4us
I20260812 19:13:51.145469 21287 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:51.114149 24559 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:51.124372 24562 raft_consensus.cc:2726] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7.
I20260812 19:13:51.130023 24552 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:51.180038 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:51.145646 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:51.122119 24456 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:51.193181 22048 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:13:51.195089 24456 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:51.175390 24433 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:51.196731 24433 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:51.198789 21114 leader_election.cc:334] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 1.535s (SENT)
I20260812 19:13:51.200436 24552 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:51.203235 24564 log.cc:924] Time spent T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.182s user 0.000s sys 0.000s
I20260812 19:13:51.203748 24571 raft_consensus.cc:2726] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:51.151669 24292 raft_consensus.cc:2726] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:51.139708 23750 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
W20260812 19:13:51.208039 24558 log.cc:924] Time spent T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.178s user 0.001s sys 0.000s
I20260812 19:13:51.207927 23750 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:51.209811 24418 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:51.210546 24418 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:51.207825 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:51.213506 24418 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:51.138994 24576 raft_consensus.cc:2781] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:51.214500 24576 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:51.214993 24576 raft_consensus.cc:3037] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:51.203619 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:51.216490 24557 log.cc:924] Time spent T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.181s user 0.000s sys 0.000s
W20260812 19:13:51.250090 21114 leader_election.cc:334] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 1.580s (SENT)
W20260812 19:13:51.268936 24584 log.cc:924] Time spent T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
I20260812 19:13:51.188675 24508 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:51.215792 24563 log.cc:924] Time spent T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.215s user 0.000s sys 0.001s
I20260812 19:13:51.217743 24513 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:51.215324 24453 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:51.159145 24559 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:51.217604 24524 consensus_queue.cc:237] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:51.215153 23750 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:51.313868 24420 consensus_meta.cc:287] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.184s user 0.009s sys 0.014s
I20260812 19:13:51.320770 23425 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:51.323004 24560 log.cc:924] Time spent T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.439s user 0.000s sys 0.002s
I20260812 19:13:51.375630 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:51.376574 21329 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
W20260812 19:13:51.377259 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 127248us
I20260812 19:13:51.379765 21112 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:51.377414 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:51.382244 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:51.386370 21322 raft_consensus.cc:1261] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260812 19:13:51.389451 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 244032us
W20260812 19:13:51.385005 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 239432us
I20260812 19:13:51.392521 21287 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:51.393146 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:51.398242 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:51.397447 21287 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:51.386880 24571 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:51.378583 24552 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:51.392273 23425 consensus_queue.cc:1035] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
W20260812 19:13:51.626482 21158 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1946) took 1542 ms (client timeout 1545 ms). Trace:
I20260812 19:13:51.537568 21329 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
W20260812 19:13:51.634696 21158 rpcz_store.cc:269] 0812 19:13:50.083759 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.084039 (+ 280us) service_pool.cc:227] Handling call
0812 19:13:51.606434 (+1522395us) spinlock_profiling.cc:243] Waited 43 ms on lock 0x7b5400123ed4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:51.626400 (+ 19966us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":98940288}
W20260812 19:13:51.605875 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 206277us
I20260812 19:13:51.408897 21151 raft_consensus.cc:1226] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:51.407703 21200 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:51.452059 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 46921.6us
I20260812 19:13:51.636406 21287 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:51.484401 21150 raft_consensus.cc:1261] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260812 19:13:51.455981 21114 leader_election.cc:334] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 1.634s (SENT)
I20260812 19:13:51.638474 21114 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:51.392616 24453 raft_consensus.cc:2781] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260812 19:13:51.639025 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 231364us
W20260812 19:13:51.640404 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 184453us
W20260812 19:13:51.641109 21114 leader_election.cc:334] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 1.678s (SENT)
I20260812 19:13:51.641502 21114 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:51.443208 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 45562.1us
W20260812 19:13:51.522919 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 113978us
I20260812 19:13:51.644709 21287 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:51.454764 24502 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:51.395421 24512 consensus_queue.cc:1035] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.251s
I20260812 19:13:51.645784 24574 raft_consensus.cc:2726] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:51.646104 24582 raft_consensus.cc:2726] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:51.643116 24453 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:51.649142 24453 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:51.649845 21200 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:51.650911 24540 raft_consensus.cc:2726] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260812 19:13:51.654371 24581 log.cc:924] Time spent T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.186s user 0.000s sys 0.000s
W20260812 19:13:51.659307 24526 consensus_meta.cc:287] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.821s user 0.012s sys 0.003s
I20260812 19:13:51.659467 21329 raft_consensus.cc:1261] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260812 19:13:51.660233 24592 log.cc:924] Time spent T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260812 19:13:51.661948 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:51.663775 21236 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1618) took 1467 ms. Trace:
I20260812 19:13:51.664207 21236 rpcz_store.cc:276] 0812 19:13:50.196390 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.200941 (+ 4551us) service_pool.cc:227] Handling call
0812 19:13:50.201246 (+ 305us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:50.718717 (+517471us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:51.663522 (+944805us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b54001f9354. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:51.663707 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1187638016}
I20260812 19:13:51.665261 24582 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:51.665591 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:51.665974 24582 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:51.666844 24553 raft_consensus.cc:2726] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260812 19:13:51.667727 22048 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8.
I20260812 19:13:51.643740 24571 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:51.640064 24292 raft_consensus.cc:2726] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:51.406703 24562 raft_consensus.cc:2781] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260812 19:13:51.669108 21240 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1615) took 1595 ms (client timeout 1751 ms). Trace:
I20260812 19:13:51.675603 21155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1951) took 1546 ms. Trace:
W20260812 19:13:51.692932 21240 rpcz_store.cc:269] 0812 19:13:50.073413 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.073878 (+ 465us) service_pool.cc:227] Handling call
0812 19:13:51.668883 (+1595005us) spinlock_profiling.cc:243] Waited 770 ms on lock 0x7b5400303754. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:51.669025 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1772076416}
I20260812 19:13:51.693281 21155 rpcz_store.cc:276] 0812 19:13:50.129019 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.129242 (+ 223us) service_pool.cc:227] Handling call
0812 19:13:50.152705 (+ 23463us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:50.805464 (+652759us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:51.675188 (+869724us) spinlock_profiling.cc:243] Waited 643 ms on lock 0x7b5400182754. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:51.675500 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1478059904}
I20260812 19:13:51.677155 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1622) took 1479 ms. Trace:
I20260812 19:13:51.678666 21288 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:13:51.694947 21243 rpcz_store.cc:276] 0812 19:13:50.197804 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.716942 (+519138us) service_pool.cc:227] Handling call
0812 19:13:50.717470 (+ 528us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:50.789055 (+ 71585us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:51.668314 (+879259us) spinlock_profiling.cc:243] Waited 70.6 ms on lock 0x7b5400304154. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:51.677047 (+ 8733us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":162344320}
I20260812 19:13:51.406455 24568 raft_consensus.cc:2781] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:13:51.696951 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:51.697587 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 55703.2us
I20260812 19:13:51.697744 21328 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:51.666369 21236 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
W20260812 19:13:51.839926 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 81796.2us
I20260812 19:13:51.727607 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:51.805594 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 160943us
W20260812 19:13:52.086056 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 415968us
W20260812 19:13:51.886556 21154 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 1950) took 1764 ms (client timeout 1719 ms). Trace:
W20260812 19:13:51.930378 21244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1554) took 2559 ms (client timeout 1508 ms). Trace:
I20260812 19:13:52.086760 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:52.096398 21244 rpcz_store.cc:269] 0812 19:13:49.371039 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.668540 (+297501us) service_pool.cc:227] Handling call
0812 19:13:51.930087 (+2261547us) spinlock_profiling.cc:243] Waited 732 ms on lock 0x7b54000b6b54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:51.930270 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1683824896}
W20260812 19:13:51.871577 21235 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1555) took 2500 ms (client timeout 2114 ms). Trace:
W20260812 19:13:52.097790 21235 rpcz_store.cc:269] 0812 19:13:49.371440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.726019 (+354579us) service_pool.cc:227] Handling call
0812 19:13:51.871257 (+2145238us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b5400040954. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:51.871452 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":948507392}
W20260812 19:13:51.723747 21233 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1532) took 3292 ms (client timeout 2145 ms). Trace:
I20260812 19:13:52.099150 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:52.099493 21233 rpcz_store.cc:269] 0812 19:13:48.384291 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:48.386979 (+ 2688us) service_pool.cc:227] Handling call
0812 19:13:51.676544 (+3289565us) spinlock_profiling.cc:243] Waited 2.34 s on lock 0x7b54002c3dd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:51.676733 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5375365376}
W20260812 19:13:52.090788 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 449677us
I20260812 19:13:51.876439 22052 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 519) took 2121 ms. Trace:
I20260812 19:13:52.101279 22052 rpcz_store.cc:276] 0812 19:13:49.755376 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.756476 (+ 1100us) service_pool.cc:227] Handling call
0812 19:13:49.756871 (+ 395us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:49.830994 (+ 74123us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:13:50.219893 (+388899us) log.cc:842] Serialized 14 byte log entry
0812 19:13:50.811562 (+591669us) log.cc:842] Serialized 14 byte log entry
0812 19:13:50.811772 (+ 210us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:50.811822 (+ 50us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:50.811876 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:51.875376 (+1063500us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b18003c4f60. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b57f96 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9
0812 19:13:51.876068 (+ 692us) spinlock_profiling.cc:243] Waited 73.8 ms on lock 0x7b54004e02d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:51.876123 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:51.876339 (+ 216us) inbound_call.cc:173] Queueing success response
0812 19:13:51.891599 (+ 15260us) spinlock_profiling.cc:243] Waited 3.41 ms on lock 0x7b18003c4f60. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a638 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
0812 19:13:51.941814 (+ 50215us) spinlock_profiling.cc:243] Waited 3.37 ms on lock 0x7b18003c4f60. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a638 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
Metrics: {"prepare.queue_time_us":17779,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":57,"raft.queue_time_us":2589,"raft.run_cpu_time_us":351,"raft.run_wall_time_us":7646,"spinlock_wait_cycles":193450624,"thread_start_us":572173,"threads_started":1,"wal-append.queue_time_us":775486}
I20260812 19:13:51.885463 21149 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 335) took 1502 ms. Trace:
W20260812 19:13:52.103178 21114 leader_election.cc:334] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 1.751s (SENT)
I20260812 19:13:52.103310 21149 rpcz_store.cc:276] 0812 19:13:50.382889 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.383263 (+ 374us) service_pool.cc:227] Handling call
0812 19:13:51.626399 (+1243136us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:51.626642 (+ 243us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:51.626681 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:51.626752 (+ 71us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:51.626808 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:51.626888 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:51.885233 (+258345us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b5400123ed0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:51.885385 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2830539520}
I20260812 19:13:51.908862 21153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1955) took 1171 ms. Trace:
I20260812 19:13:52.085716 21328 pending_rounds.cc:76] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412: Aborting all ops after (but not including) 3
W20260812 19:13:51.698984 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 56283.2us
I20260812 19:13:52.100385 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:51.779088 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.095741 21287 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:51.884316 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1620) took 1687 ms. Trace:
I20260812 19:13:51.939765 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:51.887385 21157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 344) took 1332 ms. Trace:
I20260812 19:13:52.100001 22056 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
W20260812 19:13:51.728863 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 100615us
I20260812 19:13:52.008131 21241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1617) took 1812 ms. Trace:
I20260812 19:13:52.007464 24532 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:51.892503 24453 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:52.105020 21153 rpcz_store.cc:276] 0812 19:13:50.737455 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.775844 (+ 38389us) service_pool.cc:227] Handling call
0812 19:13:50.776317 (+ 473us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:51.063746 (+287429us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:51.908452 (+844706us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x7b5400182254. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:51.908760 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":659469696}
I20260812 19:13:52.180992 21036 catalog_manager.cc:5495] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:13:52.076792 24528 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:52.093384 24562 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:52.106834 21287 leader_election.cc:398] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5.
I20260812 19:13:52.182150 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:52.183239 24568 raft_consensus.cc:686] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 7 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:52.142664 21236 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1557) took 1007 ms. Trace:
I20260812 19:13:51.700014 24456 raft_consensus.cc:2726] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:52.108351 21200 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:52.185824 21155 raft_consensus.cc:2454] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:51.863248 24516 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:52.106438 21239 rpcz_store.cc:276] 0812 19:13:50.197168 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.674352 (+477184us) service_pool.cc:227] Handling call
0812 19:13:50.849841 (+175489us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:51.085846 (+236005us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:51.883959 (+798113us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b54001f8e54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:51.884226 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":541382400}
I20260812 19:13:52.105443 21328 pending_rounds.cc:97] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260812 19:13:51.861282 24536 raft_consensus.cc:979] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059: : Instructing follower 93578836968649f6bfcaecc709871412 to start an election
W20260812 19:13:51.847891 24548 log.cc:924] Time spent T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 1.120s user 0.005s sys 0.005s
W20260812 19:13:52.095742 21154 rpcz_store.cc:269] 0812 19:13:50.121696 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.121919 (+ 223us) service_pool.cc:227] Handling call
0812 19:13:51.886304 (+1764385us) spinlock_profiling.cc:243] Waited 95.4 ms on lock 0x7b54000c4254. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:51.886454 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":219319040}
I20260812 19:13:52.167287 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.190675 21244 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:52.187177 21155 raft_consensus.cc:1226] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:52.106552 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1645) took 1008 ms. Trace:
I20260812 19:13:52.132855 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:52.152275 21151 raft_consensus.cc:1261] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260812 19:13:52.003774 24543 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.222s user 0.004s sys 0.011s
I20260812 19:13:52.185485 21236 rpcz_store.cc:276] 0812 19:13:51.135654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:51.698727 (+563073us) service_pool.cc:227] Handling call
0812 19:13:52.142512 (+443785us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:52.184335 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 77636.7us
I20260812 19:13:52.199450 21287 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:52.182281 21153 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:13:52.192303 21240 rpcz_store.cc:276] 0812 19:13:51.098750 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:51.694563 (+595813us) service_pool.cc:227] Handling call
0812 19:13:52.106480 (+411917us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:52.207823 21158 raft_consensus.cc:1261] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260812 19:13:52.186910 24516 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:52.186947 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:52.192221 24559 consensus_meta.cc:287] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.897s user 0.021s sys 0.013s
I20260812 19:13:51.785033 24292 raft_consensus.cc:2781] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader election won for term 8
I20260812 19:13:52.087638 24498 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:52.086835 24088 raft_consensus.cc:1050] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:13:51.889256 24341 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:51.928107 24594 log.cc:924] Time spent T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.341s user 0.005s sys 0.000s
I20260812 19:13:52.227341 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:52.228385 21114 leader_election.cc:334] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 1.786s (SENT)
I20260812 19:13:52.229457 21114 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:52.133343 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:52.107499 21157 rpcz_store.cc:276] 0812 19:13:50.554852 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.555522 (+ 670us) service_pool.cc:227] Handling call
0812 19:13:50.555881 (+ 359us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:50.727848 (+171967us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:51.887075 (+1159227us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b54000c3354. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:51.887314 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":367005440}
I20260812 19:13:52.196569 21243 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:13:52.232326 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1558) took 1096 ms. Trace:
I20260812 19:13:52.232806 21243 rpcz_store.cc:276] 0812 19:13:51.135953 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:51.696249 (+560296us) service_pool.cc:227] Handling call
0812 19:13:52.232257 (+536008us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:52.233990 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:52.197135 21235 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1648) took 1037 ms. Trace:
I20260812 19:13:52.235062 21235 rpcz_store.cc:276] 0812 19:13:51.159642 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.098667 (+939025us) service_pool.cc:227] Handling call
0812 19:13:52.197052 (+ 98385us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:52.236302 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:51.808934 24576 consensus_meta.cc:287] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.533s user 0.010s sys 0.000s
I20260812 19:13:52.192778 21244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1644) took 1123 ms. Trace:
W20260812 19:13:52.208267 24543 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.431s user 0.005s sys 0.014s
I20260812 19:13:51.806905 24571 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:52.249334 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:52.255646 22056 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:13:52.212334 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1563) took 1069 ms. Trace:
I20260812 19:13:52.256953 21238 rpcz_store.cc:276] 0812 19:13:51.143362 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.173178 (+1029816us) service_pool.cc:227] Handling call
0812 19:13:52.212259 (+ 39081us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:52.180151 21241 rpcz_store.cc:276] 0812 19:13:50.195912 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.200141 (+ 4229us) service_pool.cc:227] Handling call
0812 19:13:50.200554 (+ 413us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:50.205721 (+ 5167us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:13:50.760813 (+555092us) log.cc:842] Serialized 14 byte log entry
0812 19:13:51.030197 (+269384us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:51.805126 (+774929us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b4800190158. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52de94d7f1 00007f52de9448e7 00007f52de94477a 00007f52dea3895f 00007f52dea38627 00007f52dea4cb2c 00007f52dea4cad8 00007f52dea4ca58 00007f52dea4ca13 00007f52dea4bcbc 00007f52dd9cdbe6
0812 19:13:51.880424 (+ 75298us) spinlock_profiling.cc:243] Waited 1.45 ms on lock 0x7b2c00000238. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d8495e11 00007f52d84a5442 00007f52d9ae1c11 00007f52d9aa658a 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772
0812 19:13:51.881056 (+ 632us) log.cc:842] Serialized 26 byte log entry
0812 19:13:51.881274 (+ 218us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:52.002625 (+121351us) log.cc:842] Serialized 14 byte log entry
0812 19:13:52.007667 (+ 5042us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:52.007780 (+ 113us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:52.007837 (+ 57us) raft_consensus.cc:1671] finished
0812 19:13:52.007856 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:52.008040 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98659,"prepare.queue_time_us":626136,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":9201,"raft.queue_time_us":150413,"raft.run_cpu_time_us":6524,"raft.run_wall_time_us":1483072,"spinlock_wait_cycles":30262016,"thread_start_us":742882,"threads_started":3,"wal-append.queue_time_us":302281}
W20260812 19:13:52.258194 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 152240us
I20260812 19:13:52.260967 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.214253 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:52.215102 21153 raft_consensus.cc:595] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
W20260812 19:13:52.216797 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 63271.8us
I20260812 19:13:51.867995 24434 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:52.267783 24434 raft_consensus.cc:2932] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Committing config change with OpId 4.4: config changed from index 2 to 4, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:13:51.941118 24458 raft_consensus.cc:3032] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:51.860163 24433 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:52.221361 24600 raft_consensus.cc:2726] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:52.183341 24562 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:51.943843 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:51.944279 24365 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:52.135428 24459 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:52.107206 24461 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:52.187448 21328 raft_consensus.cc:2866] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:52.240108 21244 rpcz_store.cc:276] 0812 19:13:51.069429 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.166827 (+1097398us) service_pool.cc:227] Handling call
0812 19:13:52.192713 (+ 25886us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:52.241498 21287 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:52.276204 24458 raft_consensus.cc:729] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:52.293530 24458 consensus_queue.cc:260] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:52.294872 24459 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:52.295898 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.296280 24461 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:52.296780 21244 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
W20260812 19:13:52.297729 22008 leader_election.cc:334] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.987s (SENT)
I20260812 19:13:52.298095 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.261219 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:52.086997 24542 consensus_queue.cc:1035] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.142s
I20260812 19:13:52.271520 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:52.262668 21243 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:52.272193 21153 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:13:52.272310 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:52.272354 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:52.226763 24088 raft_consensus.cc:1054] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: Unable to promote non-voter 63db10063a7d4eb2a9cc9dc952654706: Illegal state: Leader has not yet committed an operation in its own term
I20260812 19:13:52.137466 24567 raft_consensus.cc:2781] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:52.239189 24579 raft_consensus.cc:1067] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Signalling peer 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:13:52.242846 24543 raft_consensus.cc:686] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:52.243808 24576 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:52.298557 24461 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:52.289368 24365 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:52.322748 24594 log.cc:924] Time spent T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20260812 19:13:52.300050 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.324240 24535 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:52.324930 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:52.290932 24531 raft_consensus.cc:2781] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:52.301398 21112 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:52.329367 24588 raft_consensus.cc:2781] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:13:52.301651 24456 consensus_queue.cc:1035] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260812 19:13:52.302404 21240 raft_consensus.cc:2379] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 6: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:52.225220 24559 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:52.334069 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1647) took 1197 ms. Trace:
I20260812 19:13:52.334285 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:52.334458 21240 rpcz_store.cc:276] 0812 19:13:51.136819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.228822 (+1092003us) service_pool.cc:227] Handling call
0812 19:13:52.334000 (+105178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:52.334985 21327 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:13:52.335727 24559 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:52.307063 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:52.336412 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:52.239965 24292 raft_consensus.cc:686] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 8 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:52.280150 21036 catalog_manager.cc:5495] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 1 to 3, leader changed from 93578836968649f6bfcaecc709871412 (127.19.70.131) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:13:52.306737 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.335464 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.198055 24532 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:52.286363 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:52.188782 24568 consensus_queue.cc:237] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:52.213335 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:52.333285 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:52.335425 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:52.314673 21329 raft_consensus.cc:2379] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 4: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:52.306133 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.305228 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.350858 21241 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:13:52.351554 21241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1560) took 1215 ms. Trace:
I20260812 19:13:52.351949 21241 rpcz_store.cc:276] 0812 19:13:51.136535 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.304796 (+1168261us) service_pool.cc:227] Handling call
0812 19:13:52.351472 (+ 46676us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:52.307150 21235 raft_consensus.cc:2454] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
W20260812 19:13:52.340260 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 98887.1us
I20260812 19:13:52.340474 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1641) took 1342 ms. Trace:
I20260812 19:13:52.353216 21240 rpcz_store.cc:276] 0812 19:13:50.998170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.335234 (+1337064us) service_pool.cc:227] Handling call
0812 19:13:52.340412 (+ 5178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:52.353713 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.354166 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.354689 21235 raft_consensus.cc:2454] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:52.354733 21287 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:52.356149 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.357705 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.338838 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:52.305799 21153 raft_consensus.cc:595] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:13:52.183326 24528 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:52.345234 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.345733 21319 raft_consensus.cc:2454] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:52.359830 21236 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:52.261618 24538 raft_consensus.cc:2726] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:52.346908 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:52.363162 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:52.360966 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.361270 21235 raft_consensus.cc:2379] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:52.363806 21320 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:52.366935 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.311707 24543 consensus_queue.cc:237] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:52.367790 21244 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:13:52.368484 21240 raft_consensus.cc:1261] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260812 19:13:52.368960 21244 raft_consensus.cc:1226] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:52.369256 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1626) took 2158 ms. Trace:
I20260812 19:13:52.369618 21244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1627) took 2143 ms. Trace:
I20260812 19:13:52.369673 21240 rpcz_store.cc:276] 0812 19:13:50.210547 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.367736 (+2157189us) service_pool.cc:227] Handling call
0812 19:13:52.368093 (+ 357us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:52.368921 (+ 828us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:52.369157 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:52.339363 24573 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:52.314332 24576 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:52.299930 22045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40624 (request call id 525) took 1208 ms. Trace:
I20260812 19:13:52.371229 24573 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:52.371507 22045 rpcz_store.cc:276] 0812 19:13:51.091806 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:51.105135 (+ 13329us) service_pool.cc:227] Handling call
0812 19:13:51.943584 (+838449us) spinlock_profiling.cc:243] Waited 4.93 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:13:52.299817 (+356233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11343232}
I20260812 19:13:52.307653 24567 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:52.372661 24528 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:52.373003 24567 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:52.373875 24573 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:52.300210 24453 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:52.329988 24588 raft_consensus.cc:686] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:52.376215 24588 consensus_queue.cc:237] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:52.311882 24571 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:52.377966 24571 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:52.374787 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260812 19:13:52.328253 24531 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:52.379853 24531 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:52.362311 24559 raft_consensus.cc:2781] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:52.381882 24559 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:52.382359 24559 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:52.324958 24535 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:52.385569 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:52.390247 22045 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 4.
I20260812 19:13:52.388909 21244 rpcz_store.cc:276] 0812 19:13:50.226655 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.368460 (+2141805us) service_pool.cc:227] Handling call
0812 19:13:52.368787 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:52.369384 (+ 597us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:52.369563 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:52.392743 21287 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:52.394650 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1701) took 1189 ms. Trace:
I20260812 19:13:52.395776 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.396615 21236 raft_consensus.cc:1261] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260812 19:13:52.387826 21243 raft_consensus.cc:1226] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:52.399951 21320 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:52.395205 21323 rpcz_store.cc:276] 0812 19:13:51.205208 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:51.206633 (+ 1425us) service_pool.cc:227] Handling call
0812 19:13:51.207028 (+ 395us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:51.291537 (+ 84509us) raft_consensus.cc:1520] Early marking committed up to index 2
0812 19:13:51.292040 (+ 503us) log.cc:842] Serialized 14 byte log entry
0812 19:13:51.391821 (+ 99781us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:51.398135 (+ 6314us) log.cc:842] Serialized 26 byte log entry
0812 19:13:51.398287 (+ 152us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:51.398319 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:51.398393 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:51.869783 (+471390us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b48000fc0d8. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52de94d7f1 00007f52de9448e7 00007f52de94477a 00007f52dea3895f 00007f52dea38627 00007f52dea4cb2c 00007f52dea4cad8 00007f52dea4ca58 00007f52dea4ca13 00007f52dea4bcbc 00007f52dd9cdbe6
0812 19:13:51.925992 (+ 56209us) spinlock_profiling.cc:243] Waited 36.7 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:52.394284 (+468292us) raft_consensus.cc:1671] finished
0812 19:13:52.394315 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:52.394507 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":76753,"prepare.queue_time_us":1013,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":64,"raft.queue_time_us":77280,"raft.run_cpu_time_us":12488,"raft.run_wall_time_us":578381,"spinlock_wait_cycles":110054912,"thread_start_us":99208,"threads_started":1,"wal-append.queue_time_us":294821}
I20260812 19:13:52.401827 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:52.401911 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.404075 21287 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:52.404388 21327 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 9.
I20260812 19:13:52.405691 21241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1562) took 1268 ms. Trace:
I20260812 19:13:52.406314 21241 rpcz_store.cc:276] 0812 19:13:51.137101 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.395466 (+1258365us) service_pool.cc:227] Handling call
0812 19:13:52.405606 (+ 10140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:52.403023 21323 raft_consensus.cc:2454] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:52.408663 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37894 (request call id 1556) took 1264 ms. Trace:
I20260812 19:13:52.408965 21244 raft_consensus.cc:1261] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260812 19:13:52.409114 21243 rpcz_store.cc:276] 0812 19:13:51.135319 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.387311 (+1251992us) service_pool.cc:227] Handling call
0812 19:13:52.387641 (+ 330us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:52.399740 (+ 12099us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:52.399949 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:52.410892 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 167336us
W20260812 19:13:52.416301 21114 leader_election.cc:334] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 1.897s (SENT)
I20260812 19:13:52.418036 21114 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:52.420415 24487 raft_consensus.cc:2726] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:52.422108 21323 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf"
dest_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:51568
I20260812 19:13:52.419643 21203 leader_election.cc:302] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:52.437323 24461 raft_consensus.cc:2726] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:13:52.385032 24535 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:52.384735 24571 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:52.388278 24292 consensus_queue.cc:237] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:52.566123 22008 leader_election.cc:334] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.583s (SENT)
I20260812 19:13:52.577071 22008 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:52.493728 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 79095.2us
W20260812 19:13:52.579408 21114 leader_election.cc:334] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 2.036s (SENT)
I20260812 19:13:52.579885 21114 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:52.580133 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.580960 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.581647 21235 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:52.581871 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:52.582491 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 164253us
W20260812 19:13:52.442728 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 334429us
I20260812 19:13:52.573101 21319 raft_consensus.cc:1226] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:52.582880 21241 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:52.594089 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.594209 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:52.596699 21241 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:52.598958 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.415223 24576 raft_consensus.cc:2726] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:52.600678 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.565364 24570 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:52.583233 24500 raft_consensus.cc:2726] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 19:13:52.611140 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 399874us
I20260812 19:13:52.605233 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.612262 21243 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:52.534195 24585 raft_consensus.cc:2726] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:52.613057 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:52.612262 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:52.613996 21239 raft_consensus.cc:1261] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260812 19:13:52.621939 21112 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:52.618940 21114 leader_election.cc:334] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 2.071s (SENT)
I20260812 19:13:52.492357 24506 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:52.633824 24506 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:52.640859 24506 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:52.643363 24516 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:52.627805 21114 leader_election.cc:334] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 2.103s (SENT)
I20260812 19:13:52.618854 24603 consensus_queue.cc:1046] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260812 19:13:52.632920 24582 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:52.645915 24516 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:52.646778 21233 raft_consensus.cc:2354] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6.
I20260812 19:13:52.646026 21114 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:52.647996 24513 raft_consensus.cc:2726] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:52.648931 24513 raft_consensus.cc:2726] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:52.653743 24365 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:52.657549 24365 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:52.660641 21318 raft_consensus.cc:1226] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:52.644775 21236 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1653) took 1013 ms. Trace:
I20260812 19:13:52.667517 21236 rpcz_store.cc:276] 0812 19:13:51.631304 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.389491 (+758187us) service_pool.cc:227] Handling call
0812 19:13:52.395480 (+ 5989us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:52.397000 (+ 1520us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:52.644437 (+247437us) spinlock_profiling.cc:243] Waited 5.63 ms on lock 0x7b54002fe750. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:52.644683 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12942208}
I20260812 19:13:52.583807 24535 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:52.671854 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 50198.6us
W20260812 19:13:52.662505 22008 leader_election.cc:334] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.024s (SENT)
I20260812 19:13:52.676795 21152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:54786 (request call id 390) took 2679 ms. Trace:
I20260812 19:13:52.673349 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:52.677932 22052 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEARNER]: Advancing to term 5
I20260812 19:13:52.602278 23432 raft_consensus.cc:2726] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 19:13:52.679059 21114 leader_election.cc:334] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 2.161s (SENT)
I20260812 19:13:52.677400 21152 rpcz_store.cc:276] 0812 19:13:49.985628 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:49.985854 (+ 226us) service_pool.cc:227] Handling call
0812 19:13:52.663760 (+2677906us) spinlock_profiling.cc:243] Waited 2.36 s on lock 0x7b5400146ed4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2baf0 00007f52d9aaef2a 00007f52de9ebc6e 00007f52d80ee807 00007f52d80ee792 00007f52d80ee6c1 00007f52d80ee63c 00007f52d80ed8f2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:52.664516 (+ 756us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5427422976}
I20260812 19:13:52.684329 21112 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:52.684588 21114 leader_election.cc:334] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 2.438s (SENT)
I20260812 19:13:52.660108 24593 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:52.660637 24500 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:52.717931 22052 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEARNER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:52.681173 24298 raft_consensus.cc:2781] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election won for term 4
W20260812 19:13:52.741154 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 75375.3us
I20260812 19:13:52.611594 24570 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:52.746008 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 123560us
W20260812 19:13:52.764276 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 369200us
W20260812 19:13:52.765128 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 361157us
I20260812 19:13:52.771941 21287 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
W20260812 19:13:52.765344 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 81102.7us
I20260812 19:13:52.773437 21112 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:52.773592 24465 consensus_queue.cc:1035] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:52.774492 24609 raft_consensus.cc:3032] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:52.775010 24609 raft_consensus.cc:729] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:52.776041 24609 consensus_queue.cc:260] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:52.777542 24609 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:52.780942 24341 raft_consensus.cc:2781] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader election won for term 9
I20260812 19:13:52.696245 24535 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:52.784931 24535 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:52.681696 24562 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:52.787557 21321 raft_consensus.cc:1226] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 6. Current term is 7. Ops: []
I20260812 19:13:52.796218 21323 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:13:52.796919 21323 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 8
W20260812 19:13:52.799958 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 202721us
I20260812 19:13:52.801944 21200 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:52.802877 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 98968.7us
I20260812 19:13:52.803205 21200 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:52.810910 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:52.811702 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 209714us
I20260812 19:13:52.742571 24532 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:52.815251 21287 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:52.811848 22053 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:52.800503 24420 raft_consensus.cc:2781] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:52.841207 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:52.842118 22052 raft_consensus.cc:3037] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:52.801822 24298 raft_consensus.cc:686] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:52.802381 24618 raft_consensus.cc:2726] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:52.811539 22045 raft_consensus.cc:1261] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260812 19:13:52.807181 24341 raft_consensus.cc:686] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 9 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:52.853284 22053 pending_rounds.cc:76] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Aborting all ops after (but not including) 3
I20260812 19:13:52.875666 22053 pending_rounds.cc:97] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260812 19:13:52.875996 22053 raft_consensus.cc:2866] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 LEARNER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:52.811975 24535 raft_consensus.cc:2781] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:52.855633 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:52.818241 24603 raft_consensus.cc:2726] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260812 19:13:52.864527 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 57116us
I20260812 19:13:52.766886 24500 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:52.882021 22052 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:52.887213 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:52.841058 24420 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:52.892094 21287 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:52.820827 24476 consensus_queue.cc:1035] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.060s
I20260812 19:13:52.823427 24573 consensus_queue.cc:1046] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260812 19:13:52.922284 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:52.855958 24626 raft_consensus.cc:2726] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:52.766546 24593 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:52.880892 24453 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:52.880438 24535 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:52.847077 24513 consensus_queue.cc:1035] T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.052s
W20260812 19:13:52.937912 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 45940.8us
I20260812 19:13:52.890388 24562 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:52.903625 24420 raft_consensus.cc:3037] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:52.847859 24616 consensus_queue.cc:1046] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.073s
I20260812 19:13:52.863080 24418 consensus_queue.cc:1046] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.071s
I20260812 19:13:52.904100 24618 raft_consensus.cc:2726] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:52.871322 24542 consensus_queue.cc:1046] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20260812 19:13:52.883531 24607 consensus_queue.cc:1046] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
I20260812 19:13:52.944269 24535 raft_consensus.cc:3037] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:52.951517 24453 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:52.982596 21287 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:52.984292 21287 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
W20260812 19:13:52.977556 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 135732us
I20260812 19:13:52.992681 21287 leader_election.cc:302] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:52.994437 21287 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:53.005612 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:52.968461 24570 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:53.012889 21287 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:53.017444 21114 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:53.018947 21241 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
I20260812 19:13:53.018905 21114 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:53.019260 21287 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:53.029385 24582 raft_consensus.cc:2781] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:53.030022 24582 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:53.030508 24582 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:53.032332 24637 consensus_queue.cc:1046] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 19:13:53.033595 24637 consensus_queue.cc:1046] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:13:52.997885 24562 raft_consensus.cc:2726] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:53.018811 24553 raft_consensus.cc:3032] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Stepping down as leader of term 2
I20260812 19:13:53.035133 24553 raft_consensus.cc:729] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:13:53.035501 24624 log.cc:924] Time spent T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260812 19:13:52.996647 24618 raft_consensus.cc:2741] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260812 19:13:53.036077 24553 consensus_queue.cc:260] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:53.035501 24576 raft_consensus.cc:2781] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:53.037477 24576 raft_consensus.cc:686] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 3 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:53.037642 24553 raft_consensus.cc:3037] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:53.038553 24576 consensus_queue.cc:237] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:53.039546 24622 consensus_queue.cc:1035] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.088s
I20260812 19:13:52.974196 24298 consensus_queue.cc:237] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:53.035856 24500 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:53.039078 24453 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:52.997334 24626 raft_consensus.cc:2726] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:52.978564 24341 consensus_queue.cc:237] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:52.940802 24629 consensus_queue.cc:1046] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.076s
I20260812 19:13:52.967108 24516 consensus_queue.cc:1035] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.044s
W20260812 19:13:53.081351 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 62174.3us
I20260812 19:13:53.129369 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:53.132426 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:53.132620 21287 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:53.133373 21153 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:13:53.011245 24619 raft_consensus.cc:2781] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:53.134404 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:53.136596 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 264246us
W20260812 19:13:53.134480 22008 leader_election.cc:334] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.437s (SENT)
I20260812 19:13:53.140518 21287 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
W20260812 19:13:53.138644 22011 leader_election.cc:334] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 1.987s (SENT)
I20260812 19:13:53.142562 22011 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:53.148365 24500 raft_consensus.cc:2726] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:53.157174 24500 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:53.157970 24500 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:53.161185 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:53.162180 21328 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:53.165481 22011 leader_election.cc:334] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 2.024s (SENT)
I20260812 19:13:53.168704 22011 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:53.171259 21287 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:53.172771 21287 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:53.050737 24567 consensus_meta.cc:287] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.677s user 0.013s sys 0.000s
W20260812 19:13:53.173357 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 167126us
I20260812 19:13:53.177297 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:53.177651 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 158835us
I20260812 19:13:53.178308 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:52.979460 24573 consensus_queue.cc:1035] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.152s
W20260812 19:13:53.048357 24617 log.cc:924] Time spent T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.071s user 0.002s sys 0.000s
W20260812 19:13:53.181227 22011 leader_election.cc:334] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 2.187s (SENT)
I20260812 19:13:53.183431 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:53.183907 22011 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:53.184206 21157 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:53.184873 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:53.045156 24453 raft_consensus.cc:3032] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:13:53.052186 24637 raft_consensus.cc:2726] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260812 19:13:53.042563 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:53.047539 24638 raft_consensus.cc:3032] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:53.227849 21151 raft_consensus.cc:1226] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:53.228636 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:53.230975 21241 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:53.175292 24626 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:53.231750 21204 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:53.162416 24619 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:53.178824 24562 raft_consensus.cc:2726] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:53.239789 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:13:53.178742 24624 log.cc:924] Time spent T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.124s user 0.000s sys 0.001s
I20260812 19:13:53.044390 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:53.239185 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:53.201339 24618 raft_consensus.cc:2726] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:53.248667 21204 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:53.265650 21154 raft_consensus.cc:1226] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
I20260812 19:13:53.253010 21151 raft_consensus.cc:1226] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:53.255185 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:53.258365 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:53.267046 21203 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:53.259132 24612 raft_consensus.cc:2726] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260812 19:13:53.239773 24619 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:13:53.259763 24531 consensus_meta.cc:287] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.879s user 0.013s sys 0.000s
W20260812 19:13:53.260295 24633 log.cc:924] Time spent T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260812 19:13:53.260412 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:53.212397 24500 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:53.261364 24634 log.cc:924] Time spent T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260812 19:13:53.209263 24453 raft_consensus.cc:729] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:53.258515 24292 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:53.269680 21241 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:53.270740 21150 raft_consensus.cc:1226] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:53.272056 21203 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:53.322201 21204 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:53.240255 24576 raft_consensus.cc:2781] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election won for term 6
W20260812 19:13:53.393260 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 121301us
I20260812 19:13:53.229610 24638 raft_consensus.cc:729] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:53.251834 24418 raft_consensus.cc:2726] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:13:53.235034 24644 raft_consensus.cc:2726] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:13:53.332159 24618 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:53.207815 24567 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:53.403230 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 220070us
W20260812 19:13:53.399484 21323 consensus_meta.cc:287] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.602s user 0.010s sys 0.002s
I20260812 19:13:53.414690 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1724) took 1027 ms. Trace:
I20260812 19:13:53.427211 21114 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:53.393566 21328 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:53.436774 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:53.426680 21323 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:53.426730 21329 rpcz_store.cc:276] 0812 19:13:52.387264 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.638449 (+251185us) service_pool.cc:227] Handling call
0812 19:13:52.669587 (+ 31138us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:52.673313 (+ 3726us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:52.673347 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:52.673848 (+ 501us) log.cc:842] Serialized 26 byte log entry
0812 19:13:52.921572 (+247724us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:52.921786 (+ 214us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:52.921897 (+ 111us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:53.414335 (+492438us) raft_consensus.cc:1671] finished
0812 19:13:53.414365 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:53.414575 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3249,"raft.queue_time_us":4357,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":5366,"thread_start_us":247329,"threads_started":1,"wal-append.queue_time_us":314754}
I20260812 19:13:53.337150 24531 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:53.473331 21327 raft_consensus.cc:1226] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:53.475958 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:53.421682 24559 consensus_meta.cc:287] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 1.039s user 0.010s sys 0.004s
W20260812 19:13:53.420629 24625 log.cc:924] Time spent T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.109s user 0.000s sys 0.001s
I20260812 19:13:53.391420 24574 raft_consensus.cc:2726] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:53.413519 24621 raft_consensus.cc:3032] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Stepping down as leader of term 7
I20260812 19:13:53.452154 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1723) took 1065 ms. Trace:
W20260812 19:13:53.482097 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 53974.3us
I20260812 19:13:53.411844 24542 raft_consensus.cc:2726] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:53.409610 24576 raft_consensus.cc:686] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 6 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:53.423843 24616 raft_consensus.cc:2781] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:53.421190 24651 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:53.499532 21319 rpcz_store.cc:276] 0812 19:13:52.386949 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.586019 (+199070us) service_pool.cc:227] Handling call
0812 19:13:52.616868 (+ 30849us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:52.617178 (+ 310us) raft_consensus.cc:1520] Early marking committed up to index 4
0812 19:13:52.888788 (+271610us) log.cc:842] Serialized 14 byte log entry
0812 19:13:53.443158 (+554370us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:53.443823 (+ 665us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:53.444631 (+ 808us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:53.445598 (+ 967us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:53.452054 (+ 6456us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":352322,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":11946,"raft.queue_time_us":49732,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":4620,"thread_start_us":763235,"threads_started":3}
I20260812 19:13:53.501333 21114 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:53.485772 24434 raft_consensus.cc:2726] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:53.497412 24640 raft_consensus.cc:2726] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260812 19:13:53.498520 24621 raft_consensus.cc:729] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:53.395467 24292 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:53.508488 24616 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:53.413944 24618 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:53.449386 24579 consensus_queue.cc:1159] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "63db10063a7d4eb2a9cc9dc952654706" responder_term: 3 status { last_received { term: 3 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 3 index: 5 } }
I20260812 19:13:53.530596 21114 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:53.541054 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 218968us
W20260812 19:13:53.542325 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 144635us
I20260812 19:13:53.470800 24453 consensus_queue.cc:260] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:53.497176 24559 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:53.597886 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 67399.6us
W20260812 19:13:53.644129 22056 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.743s user 0.007s sys 0.009s
W20260812 19:13:53.644659 22056 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.744s user 0.007s sys 0.009s
I20260812 19:13:53.644703 21114 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
W20260812 19:13:53.595255 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 51762.8us
I20260812 19:13:53.605863 21112 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:53.531388 24616 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:13:53.631770 24636 log.cc:924] Time spent T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.230s user 0.000s sys 0.001s
I20260812 19:13:53.515362 24542 consensus_queue.cc:1046] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260812 19:13:53.646631 21114 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:53.650599 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 53027.6us
I20260812 19:13:53.658198 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:53.542078 24573 raft_consensus.cc:2781] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:53.666200 22011 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:53.692438 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:53.699088 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:53.700089 21320 raft_consensus.cc:1226] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
W20260812 19:13:53.703608 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 47311.1us
I20260812 19:13:53.699980 21244 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:53.518772 24638 consensus_queue.cc:260] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:53.694181 24609 consensus_meta.cc:287] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.916s user 0.007s sys 0.009s
I20260812 19:13:53.705538 24574 consensus_queue.cc:1046] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260812 19:13:53.601142 24649 raft_consensus.cc:2726] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260812 19:13:53.690860 24573 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:53.728804 24573 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:53.728860 24649 raft_consensus.cc:2726] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:13:53.715646 24612 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:53.730046 24612 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:53.730569 24612 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Advancing to term 7
W20260812 19:13:53.699708 24655 log.cc:924] Time spent T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.168s user 0.000s sys 0.001s
I20260812 19:13:53.699684 24456 raft_consensus.cc:2781] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260812 19:13:53.715238 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 109510us
W20260812 19:13:53.774195 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 55026.8us
W20260812 19:13:53.741725 21287 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.008s sys 0.002s
W20260812 19:13:53.775518 21287 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.602s user 0.008s sys 0.003s
W20260812 19:13:53.775880 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 603165us
I20260812 19:13:53.776605 21287 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:53.655934 24641 raft_consensus.cc:2741] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260812 19:13:53.517488 24651 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:53.786628 24651 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:53.787519 24651 raft_consensus.cc:1050] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:13:53.783299 24568 raft_consensus.cc:2726] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:53.787442 21287 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:53.785518 21241 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:13:53.656200 24576 consensus_queue.cc:237] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:53.919678 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 184883us
W20260812 19:13:53.959285 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 172310us
I20260812 19:13:53.911211 24656 raft_consensus.cc:2781] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election won for term 5
W20260812 19:13:53.944170 22011 leader_election.cc:334] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 1.628s (SENT)
I20260812 19:13:53.967057 22011 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:53.917824 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:53.928573 24559 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:53.963063 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:53.929775 24663 log.cc:924] Time spent T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20260812 19:13:53.886703 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1714) took 1545 ms. Trace:
W20260812 19:13:53.792847 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 146193us
I20260812 19:13:53.968439 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:53.923648 24298 raft_consensus.cc:979] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059: : Instructing follower 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:13:53.987887 21204 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:53.952739 24456 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
W20260812 19:13:53.957023 24582 consensus_meta.cc:287] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.926s user 0.013s sys 0.000s
I20260812 19:13:53.743469 24567 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:53.774272 22050 raft_consensus.cc:1261] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260812 19:13:53.820520 24562 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:54.005550 24612 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:54.005860 24562 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:54.012084 24562 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:53.977674 21114 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:54.013747 21114 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:54.015240 24674 raft_consensus.cc:1067] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Signalling peer 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:13:54.019260 21114 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
W20260812 19:13:54.026803 24643 log.cc:924] Time spent T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.538s user 0.008s sys 0.000s
I20260812 19:13:54.027347 24646 raft_consensus.cc:3032] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:13:54.027590 24642 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:54.027863 24612 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:54.028365 24642 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:53.976526 21324 rpcz_store.cc:276] 0812 19:13:52.341006 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.385658 (+ 44652us) service_pool.cc:227] Handling call
0812 19:13:52.386027 (+ 369us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:52.681757 (+295730us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:52.681911 (+ 154us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:52.693871 (+ 11960us) log.cc:842] Serialized 26 byte log entry
0812 19:13:52.970947 (+277076us) raft_consensus.cc:1631] Marking committed up to 1
0812 19:13:52.971000 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:52.972689 (+ 1689us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:53.874885 (+902196us) raft_consensus.cc:1671] finished
0812 19:13:53.886378 (+ 11493us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:53.886591 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":63380,"raft.run_cpu_time_us":320,"raft.run_wall_time_us":8398,"thread_start_us":458457,"threads_started":2,"wal-append.queue_time_us":531250}
W20260812 19:13:53.974599 22011 leader_election.cc:334] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 2.437s (SENT)
I20260812 19:13:53.986161 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:53.999949 24582 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:53.932169 24666 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:54.019299 24653 log.cc:924] Time spent T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.389s user 0.000s sys 0.001s
I20260812 19:13:54.033272 24662 raft_consensus.cc:2726] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:54.028045 24646 raft_consensus.cc:729] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:54.047803 24666 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:54.033206 24579 raft_consensus.cc:2726] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:54.057385 22046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 541) took 1391 ms. Trace:
I20260812 19:13:54.057632 24646 consensus_queue.cc:260] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:54.035043 24651 consensus_queue.cc:237] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 3, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:54.060170 24646 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:54.062124 24656 raft_consensus.cc:686] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:54.062924 24642 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:54.063618 24656 consensus_queue.cc:237] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:54.049628 24619 consensus_meta.cc:287] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.711s user 0.000s sys 0.014s
I20260812 19:13:54.067896 24619 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:54.069460 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1722) took 1683 ms. Trace:
I20260812 19:13:54.031600 24562 consensus_queue.cc:1035] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260812 19:13:54.070767 24619 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:54.078176 21323 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:53.911445 24292 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:53.929345 24618 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:53.771829 24434 consensus_queue.cc:1046] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.205s
I20260812 19:13:53.994366 21244 raft_consensus.cc:1226] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 6. Current term is 7. Ops: []
I20260812 19:13:53.972371 24568 raft_consensus.cc:2726] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 19:13:53.998278 24559 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:53.990892 24420 consensus_meta.cc:287] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.018s user 0.008s sys 0.005s
I20260812 19:13:53.996556 24456 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
W20260812 19:13:54.002602 24645 log.cc:924] Time spent T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.243s user 0.000s sys 0.001s
I20260812 19:13:53.874990 24531 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:53.663980 24621 consensus_queue.cc:260] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
W20260812 19:13:54.007342 24658 log.cc:924] Time spent T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.342s user 0.000s sys 0.001s
W20260812 19:13:54.008160 24639 log.cc:924] Time spent T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.340s user 0.001s sys 0.000s
I20260812 19:13:54.008038 21237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1616) took 3870 ms. Trace:
W20260812 19:13:54.008419 24626 consensus_meta.cc:287] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.746s user 0.010s sys 0.001s
I20260812 19:13:53.968008 24542 consensus_queue.cc:1046] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.024s
I20260812 19:13:54.032528 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.032985 22011 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:13:54.161216 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 186660us
I20260812 19:13:54.033270 22056 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:13:54.037333 22008 leader_election.cc:334] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.704s (SENT)
I20260812 19:13:54.037606 21114 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:54.079296 22048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 532) took 1981 ms. Trace:
W20260812 19:13:53.925287 24654 log.cc:924] Time spent T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260812 19:13:54.063751 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:13:54.146836 24582 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:54.172042 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:54.172991 21233 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1646) took 3052 ms (client timeout 2161 ms). Trace:
W20260812 19:13:54.173466 21233 rpcz_store.cc:269] 0812 19:13:51.120768 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.100367 (+979599us) service_pool.cc:227] Handling call
0812 19:13:54.172745 (+2072378us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b5400150ed4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:54.172913 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":938840320}
I20260812 19:13:54.174081 21240 raft_consensus.cc:1226] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:54.067014 21036 catalog_manager.cc:5495] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 1 to 5, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 5 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:13:54.134032 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1731) took 1343 ms. Trace:
I20260812 19:13:54.120877 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:51568 (request call id 1721) took 1703 ms. Trace:
I20260812 19:13:54.176530 21323 rpcz_store.cc:276] 0812 19:13:52.386146 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.421907 (+ 35761us) service_pool.cc:227] Handling call
0812 19:13:54.076717 (+1654810us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b540016e9d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9a9dda7 00007f52de9e9ee0 00007f52d80e7784 00007f52d80e7712 00007f52d80e7641 00007f52d80e75bc 00007f52d80e6872 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:54.089452 (+ 12735us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":833047936}
W20260812 19:13:54.140486 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 142153us
I20260812 19:13:54.057938 22046 rpcz_store.cc:276] 0812 19:13:52.666394 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.667032 (+ 638us) service_pool.cc:227] Handling call
0812 19:13:54.056474 (+1389442us) spinlock_profiling.cc:243] Waited 5.07 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:54.056746 (+ 272us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:54.056969 (+ 223us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:54.057002 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:54.057057 (+ 55us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:54.057091 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.057156 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:54.057307 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11660032}
I20260812 19:13:54.163115 22008 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:54.153570 21237 rpcz_store.cc:276] 0812 19:13:50.137699 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:50.137840 (+ 141us) service_pool.cc:227] Handling call
0812 19:13:50.170504 (+ 32664us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:50.397311 (+226807us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:50.397359 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:51.901383 (+1504024us) spinlock_profiling.cc:243] Waited 6.09 ms on lock 0x7b2c00000238. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d8495e11 00007f52d84a5442 00007f52d9ae1c11 00007f52d9aa658a 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772
0812 19:13:51.902091 (+ 708us) log.cc:842] Serialized 52 byte log entry
0812 19:13:52.183710 (+281619us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:52.341263 (+157553us) log.cc:842] Serialized 14 byte log entry
0812 19:13:52.400594 (+ 59331us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:52.439504 (+ 38910us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:54.003005 (+1563501us) spinlock_profiling.cc:243] Waited 4.67 ms on lock 0x7b180033b6e0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a638 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
0812 19:13:54.007795 (+ 4790us) raft_consensus.cc:1671] finished
0812 19:13:54.007815 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:54.007967 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5164,"prepare.queue_time_us":81252,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":56,"raft.queue_time_us":210695,"raft.run_cpu_time_us":386,"raft.run_wall_time_us":43024,"spinlock_wait_cycles":24752256,"thread_start_us":659082,"threads_started":3,"wal-append.queue_time_us":359981}
I20260812 19:13:54.142059 24683 raft_consensus.cc:2781] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:54.070046 21327 rpcz_store.cc:276] 0812 19:13:52.386557 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.434679 (+ 48122us) service_pool.cc:227] Handling call
0812 19:13:52.435123 (+ 444us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:53.478859 (+1043736us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.069090 (+590231us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54002305d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:54.069366 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2386983808}
W20260812 19:13:54.170440 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 147880us
I20260812 19:13:54.179860 21114 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:54.170985 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:54.181088 21157 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
W20260812 19:13:54.183476 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 146184us
I20260812 19:13:54.174886 21328 rpcz_store.cc:276] 0812 19:13:52.790449 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:53.517993 (+727544us) service_pool.cc:227] Handling call
0812 19:13:53.539163 (+ 21170us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:53.539400 (+ 237us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:53.700561 (+161161us) log.cc:842] Serialized 14 byte log entry
0812 19:13:54.133407 (+432846us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:54.133511 (+ 104us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:54.133591 (+ 80us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.133691 (+ 100us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:54.133909 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":469291,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":141,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":132,"thread_start_us":592823,"threads_started":2,"wal-append.queue_time_us":427218}
I20260812 19:13:54.186276 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:13:54.184644 22008 leader_election.cc:334] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.809s (SENT)
I20260812 19:13:54.188436 22008 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:54.188992 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.190181 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.190410 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.192145 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.193874 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.194195 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:54.197615 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.198637 21327 raft_consensus.cc:2454] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:54.200358 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:54.188838 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:54.202265 21150 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
W20260812 19:13:54.203002 21287 connection.cc:232] RPC call timeout handler was delayed by -2.1738802449544892s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1867s for a response
I20260812 19:13:54.160553 24626 raft_consensus.cc:2726] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:54.191205 21324 raft_consensus.cc:2454] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
W20260812 19:13:54.203995 21287 connection.cc:232] RPC call timeout handler was delayed by -2.1468802449526265s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1897s for a response
I20260812 19:13:54.169384 24644 raft_consensus.cc:2726] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:54.207623 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.148437 24420 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:54.215107 21287 connection.cc:232] RPC call timeout handler was delayed by -2.1360802449053153s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19090000000000001s for a response
W20260812 19:13:54.223814 21287 connection.cc:232] RPC call timeout handler was delayed by -1.8840802449267358s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2189s for a response
W20260812 19:13:54.224190 21287 connection.cc:232] RPC call timeout handler was delayed by -1.4214802449569106s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2703s for a response
I20260812 19:13:54.216534 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:54.224886 21287 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:54.224954 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:54.209535 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:54.153703 24681 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:54.218230 21319 raft_consensus.cc:1226] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:54.219537 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.227741 21328 raft_consensus.cc:2396] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260812 19:13:54.221578 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:54.217341 21323 raft_consensus.cc:2454] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:54.231400 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.232375 21327 raft_consensus.cc:2454] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:13:54.182898 24568 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:54.154204 24559 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:54.183307 24683 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:54.247123 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:54.176144 24270 consensus_queue.cc:1035] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260812 19:13:54.212922 24582 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:54.269543 24603 raft_consensus.cc:3032] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:54.497179 24582 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:54.292027 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 106938us
I20260812 19:13:54.499998 24582 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:54.500329 24459 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:54.500120 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:54.494818 24453 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:54.502084 21287 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
W20260812 19:13:54.547904 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 45908us
I20260812 19:13:54.548605 24516 raft_consensus.cc:3032] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:54.257668 21149 raft_consensus.cc:1226] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:54.503515 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.503532 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.497771 24603 raft_consensus.cc:729] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:13:54.432402 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 247771us
I20260812 19:13:54.353025 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:54.269449 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1712) took 1934 ms. Trace:
I20260812 19:13:54.257500 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.549007 21287 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:54.506666 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:54.352010 24573 consensus_meta.cc:287] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.623s user 0.019s sys 0.005s
I20260812 19:13:54.268363 22051 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 685) took 1453 ms. Trace:
W20260812 19:13:54.488978 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 309597us
I20260812 19:13:54.599323 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1734) took 1377 ms. Trace:
I20260812 19:13:54.599475 24573 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:54.599778 21329 rpcz_store.cc:276] 0812 19:13:53.222060 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:53.475476 (+253416us) service_pool.cc:227] Handling call
0812 19:13:54.599103 (+1123627us) spinlock_profiling.cc:243] Waited 668 ms on lock 0x7b540023f0d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:54.599258 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1536531200}
I20260812 19:13:54.601155 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.602252 21329 raft_consensus.cc:2396] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260812 19:13:54.549153 24516 raft_consensus.cc:729] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:13:54.609678 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 60769.8us
I20260812 19:13:54.245147 24666 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:54.271071 22053 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 688) took 1448 ms. Trace:
I20260812 19:13:54.394560 24570 raft_consensus.cc:3032] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 8 LEADER]: Stepping down as leader of term 8
I20260812 19:13:54.373137 24665 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:54.593291 24678 raft_consensus.cc:2726] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:13:54.594193 21321 rpcz_store.cc:276] 0812 19:13:52.338428 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.366715 (+ 28287us) service_pool.cc:227] Handling call
0812 19:13:52.367008 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:52.795963 (+428955us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.269134 (+1473171us) spinlock_profiling.cc:243] Waited 420 ms on lock 0x7b540016e9d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:54.269373 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":966654336}
I20260812 19:13:54.594210 21157 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:54.596403 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.322788 24579 raft_consensus.cc:2726] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:13:54.398154 24695 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:54.322742 24298 consensus_queue.cc:1046] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.294s
W20260812 19:13:54.598816 24702 log.cc:924] Time spent T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.368s user 0.000s sys 0.001s
I20260812 19:13:54.611009 21287 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:54.671162 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1981) took 1862 ms. Trace:
I20260812 19:13:54.671757 21321 rpcz_store.cc:276] 0812 19:13:52.809054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.640581 (+1831527us) service_pool.cc:227] Handling call
0812 19:13:54.670405 (+ 29824us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:54.670672 (+ 267us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:54.670710 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:54.670772 (+ 62us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:54.670827 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.670903 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:54.671065 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:13:54.674018 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 63137.2us
I20260812 19:13:54.678927 24666 raft_consensus.cc:2726] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:54.679570 22047 raft_consensus.cc:1261] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEARNER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260812 19:13:54.679806 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:54.633738 24570 raft_consensus.cc:729] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:54.613075 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:54.620034 21323 raft_consensus.cc:1261] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260812 19:13:54.619731 24603 consensus_queue.cc:260] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:54.647480 24582 raft_consensus.cc:2781] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:54.698366 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1977) took 1900 ms. Trace:
I20260812 19:13:54.699591 21319 rpcz_store.cc:276] 0812 19:13:52.798065 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:53.596168 (+798103us) service_pool.cc:227] Handling call
0812 19:13:53.655020 (+ 58852us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:54.226740 (+571720us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.698082 (+471342us) spinlock_profiling.cc:243] Waited 563 ms on lock 0x7b540023ebd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:54.698305 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1294747008}
I20260812 19:13:54.245167 24644 raft_consensus.cc:2726] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260812 19:13:54.335147 24616 consensus_meta.cc:287] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.636s user 0.019s sys 0.000s
W20260812 19:13:54.275547 24535 consensus_meta.cc:287] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.273s user 0.007s sys 0.015s
I20260812 19:13:54.708376 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1982) took 1899 ms. Trace:
I20260812 19:13:54.708920 21321 rpcz_store.cc:276] 0812 19:13:52.809397 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.685859 (+1876462us) service_pool.cc:227] Handling call
0812 19:13:54.707833 (+ 21974us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:54.708020 (+ 187us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:54.708043 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:54.708083 (+ 40us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:54.708157 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.708210 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:54.708318 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:54.550069 21324 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
W20260812 19:13:54.484972 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 85195.6us
I20260812 19:13:54.619875 24516 consensus_queue.cc:260] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:54.324066 24553 consensus_meta.cc:287] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.286s user 0.013s sys 0.012s
I20260812 19:13:54.622536 22053 rpcz_store.cc:276] 0812 19:13:52.823097 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.823322 (+ 225us) service_pool.cc:227] Handling call
0812 19:13:52.853045 (+ 29723us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:53.128689 (+275644us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:53.128970 (+ 281us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:53.129588 (+ 618us) log.cc:842] Serialized 52 byte log entry
0812 19:13:53.332469 (+202881us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:53.403481 (+ 71012us) log.cc:842] Serialized 14 byte log entry
0812 19:13:54.035256 (+631775us) spinlock_profiling.cc:243] Waited 1.58 ms on lock 0x7b18002f1a60. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b014ca 00007f52e01d6f99 00007f52d69c5990 00007f52d6b87864 00007f52d6b87155 00007f52dd47b167 00007f52d9ab0807 00007f52d9abff0f 00007f52d9abfe88
0812 19:13:54.035527 (+ 271us) log.cc:842] Serialized 14 byte log entry
0812 19:13:54.035750 (+ 223us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.035827 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:54.270796 (+234969us) raft_consensus.cc:1671] finished
0812 19:13:54.270824 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:54.270993 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4287,"prepare.queue_time_us":74212,"raft.queue_time_us":39364,"raft.run_cpu_time_us":500,"raft.run_wall_time_us":29257,"spinlock_wait_cycles":11201664,"thread_start_us":194073,"threads_started":2,"wal-append.queue_time_us":313235}
I20260812 19:13:54.667198 24662 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:54.258929 24649 raft_consensus.cc:2781] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:54.641315 21327 raft_consensus.cc:2379] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
W20260812 19:13:54.641242 22008 leader_election.cc:334] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.048s (SENT)
W20260812 19:13:54.397439 24620 log.cc:924] Time spent T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 1.400s user 0.018s sys 0.008s
I20260812 19:13:54.599493 22051 rpcz_store.cc:276] 0812 19:13:52.792973 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.793107 (+ 134us) service_pool.cc:227] Handling call
0812 19:13:52.810844 (+ 17737us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:52.824423 (+ 13579us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:52.824471 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:52.836956 (+ 12485us) log.cc:842] Serialized 52 byte log entry
0812 19:13:53.002441 (+165485us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:53.002488 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:53.002627 (+ 139us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:54.246075 (+1243448us) raft_consensus.cc:1671] finished
0812 19:13:54.246105 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:54.246357 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1248,"raft.queue_time_us":14646,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":19386,"thread_start_us":79705,"threads_started":1,"wal-append.queue_time_us":543931}
I20260812 19:13:54.599027 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.724433 21328 raft_consensus.cc:2454] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 1.
I20260812 19:13:54.236826 24626 raft_consensus.cc:2726] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:54.251991 24683 raft_consensus.cc:3037] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:54.602583 24573 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:54.419795 24420 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:54.450350 24574 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:54.686340 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1979) took 1885 ms. Trace:
I20260812 19:13:54.625181 22056 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:13:54.691202 24603 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:54.705163 24674 raft_consensus.cc:2726] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:54.692373 24570 consensus_queue.cc:260] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:54.165122 22048 rpcz_store.cc:276] 0812 19:13:52.098273 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.107244 (+ 8971us) service_pool.cc:227] Handling call
0812 19:13:52.133404 (+ 26160us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:52.417475 (+284071us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:52.419883 (+ 2408us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:52.458074 (+ 38191us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:52.484893 (+ 26819us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.078971 (+1594078us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b54006a0ed4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:54.079024 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:54.079207 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":631671040}
I20260812 19:13:54.812806 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1713) took 2474 ms. Trace:
I20260812 19:13:54.813297 21322 rpcz_store.cc:276] 0812 19:13:52.338782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.385154 (+ 46372us) service_pool.cc:227] Handling call
0812 19:13:52.385553 (+ 399us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:52.418865 (+ 33312us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:52.418901 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:52.482688 (+ 63787us) log.cc:842] Serialized 26 byte log entry
0812 19:13:52.762233 (+279545us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:52.762291 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:52.762407 (+ 116us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:54.396580 (+1634173us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:54.779563 (+382983us) raft_consensus.cc:1671] finished
0812 19:13:54.779587 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:54.812726 (+ 33139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25758,"raft.queue_time_us":132441,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":37004,"spinlock_wait_cycles":24576384,"thread_start_us":225515,"threads_started":1,"wal-append.queue_time_us":452639}
I20260812 19:13:54.251606 24568 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:54.714504 22008 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:54.703024 24616 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:54.730419 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1983) took 1911 ms. Trace:
I20260812 19:13:54.818678 21319 rpcz_store.cc:276] 0812 19:13:52.819456 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.701402 (+1881946us) service_pool.cc:227] Handling call
0812 19:13:54.729730 (+ 28328us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:54.729987 (+ 257us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:54.730023 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:54.730081 (+ 58us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:54.730120 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.730188 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:54.730334 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:54.819648 21287 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:54.698344 24582 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:54.737084 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1985) took 1899 ms. Trace:
I20260812 19:13:54.822583 21324 rpcz_store.cc:276] 0812 19:13:52.837578 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.711098 (+1873520us) service_pool.cc:227] Handling call
0812 19:13:54.736554 (+ 25456us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:54.736754 (+ 200us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:54.736779 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:54.736820 (+ 41us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:54.736850 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.736906 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:54.737022 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:54.823920 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:54.824673 21322 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:54.824561 21287 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:54.737762 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1732) took 1915 ms. Trace:
I20260812 19:13:54.741258 21323 rpcz_store.cc:276] 0812 19:13:52.800809 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.604048 (+1803239us) service_pool.cc:227] Handling call
0812 19:13:54.619640 (+ 15592us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:54.685967 (+ 66327us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.686263 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:54.691860 24559 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:54.750511 21327 raft_consensus.cc:1226] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 8. Current term is 9. Ops: []
I20260812 19:13:54.751565 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:54.753468 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:54.833467 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:54.832049 21241 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:13:54.836478 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:54.754472 21328 raft_consensus.cc:1226] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:54.755688 24498 raft_consensus.cc:2726] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:54.756034 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:54.756191 24516 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:54.731052 24662 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:54.839193 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:54.768724 24652 log.cc:924] Time spent T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 1.508s user 0.002s sys 0.000s
I20260812 19:13:54.767879 24574 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:54.741215 24626 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:54.245437 24681 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:54.770973 24697 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:54.704100 24711 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:54.817654 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:54.831996 24570 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:13:54.828482 21321 rpcz_store.cc:276] 0812 19:13:52.822157 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.710106 (+1887949us) service_pool.cc:227] Handling call
0812 19:13:54.737288 (+ 27182us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:54.737460 (+ 172us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:54.737483 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:54.737522 (+ 39us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:13:54.737557 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.737609 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:54.737706 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:54.849767 24365 raft_consensus.cc:979] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059: : Instructing follower 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:13:54.762341 22045 raft_consensus.cc:1226] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEARNER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260812 19:13:54.841331 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1986) took 2000 ms. Trace:
I20260812 19:13:54.875463 21327 rpcz_store.cc:276] 0812 19:13:52.840858 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.716884 (+1876026us) service_pool.cc:227] Handling call
0812 19:13:54.750211 (+ 33327us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:54.840929 (+ 90718us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.841238 (+ 309us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:54.877956 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1736) took 1359 ms. Trace:
W20260812 19:13:54.841423 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 200218us
W20260812 19:13:54.880554 22008 leader_election.cc:334] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.952s (SENT)
I20260812 19:13:54.881201 22008 leader_election.cc:302] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:54.646588 24665 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:54.884893 24665 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:54.845501 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1987) took 1992 ms. Trace:
I20260812 19:13:54.885890 21328 rpcz_store.cc:276] 0812 19:13:52.852901 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.725259 (+1872358us) service_pool.cc:227] Handling call
0812 19:13:54.754193 (+ 28934us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:54.844915 (+ 90722us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.845408 (+ 493us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:54.888695 21328 raft_consensus.cc:1226] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:54.889534 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1737) took 1365 ms. Trace:
I20260812 19:13:54.714208 24535 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:54.892655 24535 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:54.847357 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.850523 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:54.851894 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1733) took 1643 ms. Trace:
I20260812 19:13:54.831336 24582 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:54.896739 21322 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:13:54.897429 21323 rpcz_store.cc:276] 0812 19:13:53.208288 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.828931 (+1620643us) service_pool.cc:227] Handling call
0812 19:13:54.850586 (+ 21655us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:54.851413 (+ 827us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:54.851452 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:54.851508 (+ 56us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:54.851580 (+ 72us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.851650 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:54.851817 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:54.894415 21319 raft_consensus.cc:2379] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 6: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:54.901013 21322 raft_consensus.cc:1226] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260812 19:13:54.901854 21319 raft_consensus.cc:1261] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260812 19:13:54.859162 24606 raft_consensus.cc:3032] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:13:54.889933 21328 rpcz_store.cc:276] 0812 19:13:53.524406 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.887883 (+1363477us) service_pool.cc:227] Handling call
0812 19:13:54.888410 (+ 527us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:54.889204 (+ 794us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.889459 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:54.733644 24553 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:54.723166 24649 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:54.902904 24606 raft_consensus.cc:729] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:54.905467 24649 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:54.904703 21322 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd"
dest_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:51568
I20260812 19:13:54.906351 24606 consensus_queue.cc:260] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:54.907085 21322 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:13:54.669342 24695 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:54.908403 21322 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:54.905018 24553 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:54.910086 24695 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:54.907835 24606 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:54.912139 24553 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:54.914373 24506 raft_consensus.cc:2726] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260812 19:13:54.917877 22008 leader_election.cc:334] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.107s (SENT)
I20260812 19:13:54.878470 21327 rpcz_store.cc:276] 0812 19:13:53.518953 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.876710 (+1357757us) service_pool.cc:227] Handling call
0812 19:13:54.877260 (+ 550us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:54.877514 (+ 254us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:54.877550 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:54.877610 (+ 60us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:54.877658 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.877727 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:54.877877 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:54.847108 24710 raft_consensus.cc:2726] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:54.919201 22008 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:54.857784 24538 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.004s sys 0.007s
W20260812 19:13:54.858115 24588 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.264s user 0.015s sys 0.010s
I20260812 19:13:54.934978 24365 raft_consensus.cc:1067] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Signalling peer 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:13:54.950176 24492 raft_consensus.cc:2726] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:54.855600 24626 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:54.872298 24697 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:54.998111 22045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 692) took 2051 ms. Trace:
I20260812 19:13:55.098680 22045 rpcz_store.cc:276] 0812 19:13:52.949002 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.952157 (+ 3155us) service_pool.cc:227] Handling call
0812 19:13:54.761588 (+1809431us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52b7c 00007f52d6b5273b 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:54.761848 (+ 260us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:54.937268 (+175420us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.995584 (+ 58316us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6548864}
I20260812 19:13:54.874254 24711 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:55.075598 24588 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.492s user 0.017s sys 0.012s
W20260812 19:13:55.085783 24538 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.811s user 0.010s sys 0.021s
W20260812 19:13:55.212677 24727 log.cc:924] Time spent T e06f074e35674592bdc8360835fd2661 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20260812 19:13:55.213896 24568 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:55.218657 24711 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:55.220829 24662 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:55.227039 24616 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:55.232264 24626 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:55.233489 24574 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:55.128001 24724 log.cc:924] Time spent T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.131s user 0.001s sys 0.001s
W20260812 19:13:55.237638 24456 consensus_meta.cc:287] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.079s user 0.011s sys 0.004s
I20260812 19:13:55.245901 24681 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:55.238247 24456 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:55.514526 21115 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.818s user 0.002s sys 0.001s
W20260812 19:13:55.515460 21114 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.297s user 0.008s sys 0.095s
W20260812 19:13:55.515969 21114 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.298s user 0.008s sys 0.095s
I20260812 19:13:55.517069 21151 raft_consensus.cc:1226] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
W20260812 19:13:55.515636 21115 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.820s user 0.002s sys 0.002s
I20260812 19:13:54.694253 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:54.791383 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 803538us
W20260812 19:13:55.524071 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 1.34421e+06us
I20260812 19:13:55.527906 21153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1989) took 1571 ms. Trace:
I20260812 19:13:55.531008 21114 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:54.838898 21244 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:13:55.533337 21236 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
W20260812 19:13:55.534158 21235 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.161s user 0.003s sys 0.008s
I20260812 19:13:54.778810 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:55.534746 21235 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.162s user 0.003s sys 0.009s
I20260812 19:13:55.536188 21235 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 313) took 1874 ms. Trace:
I20260812 19:13:54.849645 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 310) took 1191 ms. Trace:
I20260812 19:13:55.537328 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:54.777935 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:54.843854 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:55.539609 21237 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:13:55.537847 21239 rpcz_store.cc:276] 0812 19:13:53.658641 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:53.658810 (+ 169us) service_pool.cc:227] Handling call
0812 19:13:53.704408 (+ 45598us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:53.704708 (+ 300us) raft_consensus.cc:1520] Early marking committed up to index 7
0812 19:13:53.819393 (+114685us) log.cc:842] Serialized 14 byte log entry
0812 19:13:54.848990 (+1029597us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:54.849083 (+ 93us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:54.849187 (+ 104us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.849285 (+ 98us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:54.849574 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":276502,"prepare.queue_time_us":305238,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":406158,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":107,"thread_start_us":850317,"threads_started":3,"wal-append.queue_time_us":401031,"wal-append.run_cpu_time_us":1498,"wal-append.run_wall_time_us":1049425}
I20260812 19:13:55.541163 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:55.541002 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:55.541693 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:54.848959 21238 raft_consensus.cc:1226] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:55.542821 21241 raft_consensus.cc:2379] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 4: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
W20260812 19:13:55.547652 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 722937us
W20260812 19:13:55.548347 21329 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.851s user 0.007s sys 0.015s
W20260812 19:13:55.548966 21329 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.852s user 0.008s sys 0.016s
I20260812 19:13:55.550076 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1735) took 2033 ms. Trace:
I20260812 19:13:55.551373 24559 raft_consensus.cc:2781] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:54.929030 21322 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:55.561331 21321 rpcz_store.cc:276] 0812 19:13:53.516630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.861185 (+1344555us) service_pool.cc:227] Handling call
0812 19:13:55.548321 (+687136us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:55.549276 (+ 955us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:55.549328 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:55.549424 (+ 96us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:55.549463 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:55.549574 (+ 111us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:55.549775 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:55.562068 24559 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:55.022069 24582 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:54.945869 22008 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:55.564062 21319 raft_consensus.cc:1226] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:55.564378 21322 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:54.862825 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 314) took 1199 ms. Trace:
I20260812 19:13:55.542349 21237 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:13:55.555702 22011 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.750s user 0.009s sys 0.017s
I20260812 19:13:55.565418 21239 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:55.524165 21155 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
I20260812 19:13:55.528550 21153 rpcz_store.cc:276] 0812 19:13:53.956492 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:53.956654 (+ 162us) service_pool.cc:227] Handling call
0812 19:13:53.957036 (+ 382us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:54.164308 (+207272us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:54.164330 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:54.168911 (+ 4581us) log.cc:842] Serialized 26 byte log entry
0812 19:13:54.676418 (+507507us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:54.701013 (+ 24595us) log.cc:842] Serialized 14 byte log entry
0812 19:13:54.701227 (+ 214us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.701336 (+ 109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:55.527039 (+825703us) raft_consensus.cc:1671] finished
0812 19:13:55.527087 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:55.527557 (+ 470us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":28447,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":13160,"raft.queue_time_us":121615,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":208,"thread_start_us":660006,"threads_started":2,"wal-append.queue_time_us":678213}
W20260812 19:13:55.583736 21324 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.686s user 0.000s sys 0.003s
W20260812 19:13:55.584456 21324 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.687s user 0.000s sys 0.003s
I20260812 19:13:55.536615 21235 rpcz_store.cc:276] 0812 19:13:53.661651 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:53.661775 (+ 124us) service_pool.cc:227] Handling call
0812 19:13:53.662141 (+ 366us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:53.662458 (+ 317us) raft_consensus.cc:1520] Early marking committed up to index 6
0812 19:13:53.818840 (+156382us) log.cc:842] Serialized 14 byte log entry
0812 19:13:53.974010 (+155170us) log.cc:842] Serialized 14 byte log entry
0812 19:13:54.372723 (+398713us) spinlock_profiling.cc:243] Waited 4.58 ms on lock 0x7b18002cd880. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b014ca 00007f52e01d6f99 00007f52d69c5990 00007f52d6b87864 00007f52d6b87155 00007f52d9a9e3d0 00007f52d9ab0576 00007f52d9ac10d7 00007f52d9ac1058
0812 19:13:55.535326 (+1162603us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:55.535478 (+ 152us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:55.535641 (+ 163us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:55.535795 (+ 154us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:55.536059 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":428818,"prepare.queue_time_us":334358,"raft.queue_time_us":434150,"raft.run_cpu_time_us":366,"raft.run_wall_time_us":25155,"spinlock_wait_cycles":20744192,"thread_start_us":1410007,"threads_started":3,"wal-append.queue_time_us":171404,"wal-append.run_cpu_time_us":11812,"wal-append.run_wall_time_us":1080318}
I20260812 19:13:55.566849 24559 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:55.593407 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:51568 (request call id 1740) took 1464 ms. Trace:
I20260812 19:13:55.594707 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1708) took 3261 ms. Trace:
W20260812 19:13:55.595206 22011 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.790s user 0.010s sys 0.017s
I20260812 19:13:55.595129 21155 raft_consensus.cc:1226] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 6. Current term is 7. Ops: []
I20260812 19:13:55.595639 21156 raft_consensus.cc:1226] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 6. Current term is 7. Ops: []
I20260812 19:13:55.558887 24729 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:55.586324 21152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:54786 (request call id 391) took 2695 ms. Trace:
W20260812 19:13:55.450378 24642 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.084s sys 0.192s
I20260812 19:13:55.597824 21322 rpcz_store.cc:276] 0812 19:13:54.129031 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.904210 (+775179us) service_pool.cc:227] Handling call
0812 19:13:55.589706 (+685496us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:55.606804 24465 raft_consensus.cc:2726] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:55.607972 24553 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:55.609052 24607 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:55.608930 24506 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:55.609944 24531 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:55.610385 24623 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:55.611374 24711 raft_consensus.cc:2726] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:13:55.646736 24538 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:55.647842 24538 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:55.612634 24298 raft_consensus.cc:2781] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:13:55.649890 24298 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:55.618872 24588 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:55.612473 24420 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:55.650570 24298 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
W20260812 19:13:55.615592 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 810571us
I20260812 19:13:55.616806 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:55.650888 24588 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:55.617954 24492 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:55.653234 24492 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:55.612898 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 82423.6us
I20260812 19:13:55.654915 24729 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:55.656773 24492 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:55.614217 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1715) took 3272 ms. Trace:
I20260812 19:13:55.657173 24582 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:55.657784 21114 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:55.657568 21320 rpcz_store.cc:276] 0812 19:13:52.341276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.411671 (+ 70395us) service_pool.cc:227] Handling call
0812 19:13:52.412013 (+ 342us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:53.700666 (+1288653us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:55.613398 (+1912732us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540016d5d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:55.613944 (+ 546us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2961794304}
I20260812 19:13:55.658576 21200 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:55.660473 24616 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:55.662958 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:55.651132 24420 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:55.666255 24642 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.767s user 0.084s sys 0.194s
I20260812 19:13:55.630460 21155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 371) took 1003 ms. Trace:
I20260812 19:13:55.643584 24553 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:55.645352 24531 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:55.647889 24623 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:55.664391 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:55.670818 24506 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:55.654208 24588 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:55.663067 21158 raft_consensus.cc:1226] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:55.597921 21318 rpcz_store.cc:276] 0812 19:13:52.333467 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.334723 (+ 1256us) service_pool.cc:227] Handling call
0812 19:13:52.660395 (+325672us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:52.673162 (+ 12767us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:55.594055 (+2920893us) spinlock_profiling.cc:243] Waited 325 ms on lock 0x7b5400354b50. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:55.594419 (+ 364us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":748080896}
I20260812 19:13:55.626353 21152 rpcz_store.cc:276] 0812 19:13:52.890435 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.890795 (+ 360us) service_pool.cc:227] Handling call
0812 19:13:55.583249 (+2692454us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b5400215054. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2baf0 00007f52d9aaef2a 00007f52de9ebc6e 00007f52d80ee807 00007f52d80ee792 00007f52d80ee6c1 00007f52d80ee63c 00007f52d80ed8f2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:55.586049 (+ 2800us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32357120}
I20260812 19:13:55.576718 21243 rpcz_store.cc:276] 0812 19:13:53.663288 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:53.663476 (+ 188us) service_pool.cc:227] Handling call
0812 19:13:53.663820 (+ 344us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:53.664054 (+ 234us) raft_consensus.cc:1520] Early marking committed up to index 7
0812 19:13:53.936155 (+272101us) log.cc:842] Serialized 14 byte log entry
0812 19:13:54.862292 (+926137us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:54.862373 (+ 81us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:13:54.862445 (+ 72us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:54.862536 (+ 91us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:54.862740 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":313148,"prepare.queue_time_us":345691,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":36,"raft.queue_time_us":348878,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":5117,"thread_start_us":884003,"threads_started":3,"wal-append.queue_time_us":234849}
W20260812 19:13:55.642386 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 76646.3us
I20260812 19:13:55.658072 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:55.679101 21155 rpcz_store.cc:276] 0812 19:13:54.621229 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:55.593794 (+972565us) service_pool.cc:227] Handling call
0812 19:13:55.594758 (+ 964us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:55.622240 (+ 27482us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:55.624700 (+ 2460us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:55.661489 24616 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:55.681499 21320 raft_consensus.cc:1226] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
I20260812 19:13:55.688483 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:55.657939 24729 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:55.696349 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:55.697594 24623 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:55.698308 21235 raft_consensus.cc:2354] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4.
I20260812 19:13:55.698721 24553 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:55.704308 24607 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:55.706905 24531 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:55.707582 24628 log.cc:924] Time spent T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 2.733s user 0.000s sys 0.024s
I20260812 19:13:55.695259 21236 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8.
I20260812 19:13:55.720640 21328 raft_consensus.cc:3037] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:55.729375 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:55.730710 24697 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:55.731244 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:55.732404 21235 raft_consensus.cc:2354] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4.
I20260812 19:13:55.733157 22045 raft_consensus.cc:1261] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260812 19:13:55.733451 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:55.734726 21244 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8.
W20260812 19:13:55.735033 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 789118us
I20260812 19:13:55.738168 21322 raft_consensus.cc:1261] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260812 19:13:55.741628 24420 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:55.742100 21321 raft_consensus.cc:1226] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:55.753413 21244 raft_consensus.cc:1261] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260812 19:13:55.753643 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 67829.9us
I20260812 19:13:55.754944 24433 raft_consensus.cc:2726] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:55.762568 24456 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:55.767371 21244 raft_consensus.cc:1226] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:55.768224 24616 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:55.768523 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:55.769657 22047 raft_consensus.cc:2454] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:13:55.769745 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:55.749259 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:55.749446 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1738) took 1817 ms. Trace:
I20260812 19:13:55.771420 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:55.776404 22056 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:13:55.776446 24516 consensus_meta.cc:287] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.933s user 0.010s sys 0.010s
I20260812 19:13:55.777025 24506 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:55.777375 21233 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:13:55.771315 22048 raft_consensus.cc:2396] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260812 19:13:55.782370 22008 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:55.802793 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:55.753684 24607 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:55.804559 21328 raft_consensus.cc:2454] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:55.808398 21320 raft_consensus.cc:1226] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:55.775120 21324 rpcz_store.cc:276] 0812 19:13:53.931651 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.895582 (+963931us) service_pool.cc:227] Handling call
0812 19:13:54.896352 (+ 770us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:54.896640 (+ 288us) raft_consensus.cc:1520] Early marking committed up to index 8
0812 19:13:55.585163 (+688523us) log.cc:842] Serialized 14 byte log entry
0812 19:13:55.694838 (+109675us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:55.713889 (+ 19051us) raft_consensus.cc:1631] Marking committed up to 8
0812 19:13:55.739167 (+ 25278us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:55.743906 (+ 4739us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:55.749384 (+ 5478us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":832636,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":86,"raft.queue_time_us":51855,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":186,"thread_start_us":728218,"threads_started":2}
I20260812 19:13:55.816322 24492 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:55.821022 24492 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:55.822041 21157 consensus_meta.cc:287] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.181s user 0.014s sys 0.005s
I20260812 19:13:55.822582 21157 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
I20260812 19:13:55.822840 24516 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:55.823479 24650 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:55.823640 24516 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:55.825178 24297 raft_consensus.cc:3032] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:55.824787 24650 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:55.825644 24297 raft_consensus.cc:729] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:55.827178 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:55.828366 21244 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:55.829834 24720 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:55.832495 24720 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:55.827093 24531 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:55.835135 24531 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:55.836258 22047 raft_consensus.cc:1261] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEARNER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 19:13:55.839370 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1741) took 1680 ms. Trace:
I20260812 19:13:55.823474 21157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51004 (request call id 369) took 1471 ms. Trace:
W20260812 19:13:55.815081 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 71103.4us
I20260812 19:13:55.845762 21318 raft_consensus.cc:1226] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:55.840266 21157 rpcz_store.cc:276] 0812 19:13:54.351953 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.352620 (+ 667us) service_pool.cc:227] Handling call
0812 19:13:55.823341 (+1470721us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4864}
I20260812 19:13:55.840013 21319 rpcz_store.cc:276] 0812 19:13:54.158714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.907667 (+748953us) service_pool.cc:227] Handling call
0812 19:13:54.908021 (+ 354us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:55.564585 (+656564us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:55.838969 (+274384us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b5400355054. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:55.839275 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1507347712}
I20260812 19:13:55.844789 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1579) took 1030 ms. Trace:
I20260812 19:13:55.851581 21243 rpcz_store.cc:276] 0812 19:13:54.814469 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:55.748120 (+933651us) service_pool.cc:227] Handling call
0812 19:13:55.844717 (+ 96597us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:55.810796 24456 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:55.826435 24516 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:55.859867 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:55.817772 24538 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:55.873498 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 248011us
I20260812 19:13:55.877942 21321 raft_consensus.cc:1226] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:55.880796 24568 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:55.863862 24506 raft_consensus.cc:2726] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:55.881119 24500 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:55.886662 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 68886.7us
I20260812 19:13:55.889082 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1628) took 5659 ms. Trace:
W20260812 19:13:55.890167 24646 consensus_meta.cc:287] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.829s user 0.011s sys 0.018s
I20260812 19:13:55.890362 24664 raft_consensus.cc:3032] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:55.891147 24746 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:55.890710 24297 consensus_queue.cc:260] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:55.892592 21150 consensus_meta.cc:287] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.194s user 0.012s sys 0.007s
I20260812 19:13:55.884486 24492 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:55.893685 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 236059us
I20260812 19:13:55.883162 24568 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:55.896150 21151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1990) took 1936 ms. Trace:
I20260812 19:13:55.901782 21240 rpcz_store.cc:276] 0812 19:13:50.229907 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:52.387086 (+2157179us) service_pool.cc:227] Handling call
0812 19:13:54.173818 (+1786732us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:54.193500 (+ 19682us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:55.888578 (+1695078us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b5400150ed0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:55.888855 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4108191488}
I20260812 19:13:55.904594 24664 raft_consensus.cc:729] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:55.917866 21151 rpcz_store.cc:276] 0812 19:13:53.959309 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:53.959485 (+ 176us) service_pool.cc:227] Handling call
0812 19:13:53.959915 (+ 430us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:55.518262 (+1558347us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:55.895761 (+377499us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b5400182254. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:55.896046 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3579042688}
I20260812 19:13:55.907812 21288 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:13:55.908763 24576 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:55.909727 24657 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260812 19:13:55.881659 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:55.911597 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 129491us
I20260812 19:13:55.932721 24664 consensus_queue.cc:260] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:55.912578 21152 raft_consensus.cc:1226] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 9 LEADER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 8. Current term is 9. Ops: [8.6-8.6]
W20260812 19:13:55.914109 24735 log.cc:924] Time spent T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.801s user 0.011s sys 0.014s
I20260812 19:13:55.935660 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:55.914335 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:55.916953 21150 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:13:55.918222 21237 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:13:55.906366 24746 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:55.921625 21244 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
I20260812 19:13:55.922816 24297 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:55.958673 24629 raft_consensus.cc:3032] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:55.959142 24629 raft_consensus.cc:729] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:55.960475 21239 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:13:55.960307 24629 consensus_queue.cc:260] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:55.961154 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1577) took 1170 ms. Trace:
I20260812 19:13:55.961246 24508 raft_consensus.cc:2726] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:55.961545 21239 rpcz_store.cc:276] 0812 19:13:54.790405 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:55.541291 (+750886us) service_pool.cc:227] Handling call
0812 19:13:55.961055 (+419764us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:55.924398 24568 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:55.927088 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:55.927573 21158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54796 (request call id 1654) took 1756 ms. Trace:
I20260812 19:13:55.921064 24500 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:55.934116 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 2000) took 1156 ms. Trace:
W20260812 19:13:55.974814 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 54027.7us
I20260812 19:13:55.942072 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:55.935415 24576 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:55.943189 21156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 370) took 1590 ms. Trace:
I20260812 19:13:55.944404 24453 raft_consensus.cc:3032] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:55.944960 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:55.945577 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1990) took 1115 ms. Trace:
I20260812 19:13:55.961647 24629 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:55.948153 24746 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:55.939662 24531 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:55.949976 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:55.951892 24621 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:13:55.921144 21200 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:55.955539 24492 consensus_queue.cc:1046] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260812 19:13:55.978199 21156 rpcz_store.cc:276] 0812 19:13:54.352373 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.352546 (+ 173us) service_pool.cc:227] Handling call
0812 19:13:54.352997 (+ 451us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:55.640703 (+1287706us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:55.942740 (+302037us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b5400123ed4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:55.943095 (+ 355us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2856520960}
I20260812 19:13:55.956103 22011 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:55.957636 21236 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1584) took 1136 ms. Trace:
I20260812 19:13:55.927469 21114 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:55.966786 21237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1571) took 1150 ms. Trace:
I20260812 19:13:55.988585 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1743) took 1129 ms. Trace:
I20260812 19:13:55.934374 24664 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:13:55.973295 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 65762.8us
W20260812 19:13:55.995426 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 78763.9us
I20260812 19:13:55.995818 24492 raft_consensus.cc:3032] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:13:55.996366 24492 raft_consensus.cc:729] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:55.996654 21200 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:55.997195 24576 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:55.997547 24492 consensus_queue.cc:260] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:55.946674 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:55.975296 21320 rpcz_store.cc:276] 0812 19:13:54.777490 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:55.697325 (+919835us) service_pool.cc:227] Handling call
0812 19:13:55.807367 (+110042us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:55.808827 (+ 1460us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:55.933844 (+125017us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54002e59d0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:55.934019 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":252149632}
I20260812 19:13:55.982020 22056 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:13:56.039983 24492 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:56.044071 21233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1572) took 1255 ms. Trace:
I20260812 19:13:56.044730 24559 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:56.046941 21149 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 1978) took 2863 ms. Trace:
I20260812 19:13:55.980144 24453 raft_consensus.cc:729] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:55.978191 24657 raft_consensus.cc:2781] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:56.001101 24609 raft_consensus.cc:2726] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260812 19:13:56.001412 24500 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:56.055294 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:56.056397 21319 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:56.003682 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:55.988296 21236 rpcz_store.cc:276] 0812 19:13:54.821107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.838100 (+ 16993us) service_pool.cc:227] Handling call
0812 19:13:55.957293 (+1119193us) spinlock_profiling.cc:243] Waited 690 ms on lock 0x7b5400304154. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:55.957503 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1586483456}
W20260812 19:13:56.005162 24649 consensus_meta.cc:287] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.099s user 0.001s sys 0.019s
I20260812 19:13:56.006074 24088 consensus_queue.cc:1035] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 19:13:55.980590 21327 rpcz_store.cc:276] 0812 19:13:54.746090 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.921824 (+175734us) service_pool.cc:227] Handling call
0812 19:13:54.933845 (+ 12021us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:55.555739 (+621894us) raft_consensus.cc:1520] Early marking committed up to index 4
0812 19:13:55.628041 (+ 72302us) log.cc:842] Serialized 14 byte log entry
0812 19:13:55.812957 (+184916us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:55.813009 (+ 52us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:55.818039 (+ 5030us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:55.844028 (+ 25989us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:55.861311 (+ 17283us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17809,"prepare.queue_time_us":173515,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":18235,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":148,"thread_start_us":222076,"threads_started":2,"wal-append.queue_time_us":224375}
I20260812 19:13:56.007331 24553 raft_consensus.cc:2726] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260812 19:13:56.007397 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 80130.5us
I20260812 19:13:56.007748 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:55.972115 21158 rpcz_store.cc:276] 0812 19:13:54.170883 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.171480 (+ 597us) service_pool.cc:227] Handling call
0812 19:13:55.591611 (+1420131us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:55.713567 (+121956us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:55.927288 (+213721us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b5400182c50. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:55.927502 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3155263360}
W20260812 19:13:56.009066 24741 log.cc:924] Time spent T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
W20260812 19:13:56.012270 24603 consensus_meta.cc:287] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.256s user 0.009s sys 0.010s
I20260812 19:13:56.013353 24513 consensus_queue.cc:1046] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 19:13:56.015394 24694 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:56.015743 24458 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:56.017475 24593 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:56.055411 24657 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:56.018533 24739 log.cc:924] Time spent T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.133s user 0.001s sys 0.000s
I20260812 19:13:56.019095 24506 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:13:56.019574 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 311) took 2360 ms. Trace:
W20260812 19:13:56.020725 21150 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1661) took 1820 ms (client timeout 2152 ms). Trace:
I20260812 19:13:56.022076 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 1980) took 3217 ms. Trace:
I20260812 19:13:56.023655 24531 consensus_queue.cc:1035] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:56.023320 24650 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:56.026240 21200 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:56.080619 21238 rpcz_store.cc:276] 0812 19:13:53.659180 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:53.659338 (+ 158us) service_pool.cc:227] Handling call
0812 19:13:53.918919 (+259581us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:55.543109 (+1624190us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:56.019012 (+475903us) spinlock_profiling.cc:243] Waited 930 ms on lock 0x7b5400012cd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:56.019343 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2138370944}
I20260812 19:13:56.030134 21328 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2"
dest_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:51568
I20260812 19:13:55.917685 24456 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:56.082736 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:56.032904 24638 raft_consensus.cc:3037] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:54.168179 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:13:55.987208 24720 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:55.988935 21237 rpcz_store.cc:276] 0812 19:13:54.788463 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:55.540639 (+752176us) service_pool.cc:227] Handling call
0812 19:13:55.938464 (+397825us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:13:56.044770 21233 rpcz_store.cc:276] 0812 19:13:54.788864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:55.536583 (+747719us) service_pool.cc:227] Handling call
0812 19:13:56.043780 (+507197us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b5400043dd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:56.043976 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":475299584}
I20260812 19:13:56.002039 24465 raft_consensus.cc:2726] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:56.049772 21149 rpcz_store.cc:276] 0812 19:13:53.184968 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:53.208126 (+ 23158us) service_pool.cc:227] Handling call
0812 19:13:53.208488 (+ 362us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:54.549468 (+1340980us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:56.046174 (+1496706us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540019f0d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:13:56.046646 (+ 472us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2412282624}
I20260812 19:13:56.066107 21156 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:13:56.067312 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:55.989068 21322 rpcz_store.cc:276] 0812 19:13:54.862113 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:55.746528 (+884415us) service_pool.cc:227] Handling call
0812 19:13:55.853642 (+107114us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:55.940164 (+ 86522us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:55.953712 (+ 13548us) log.cc:842] Serialized 14 byte log entry
0812 19:13:55.988191 (+ 34479us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:55.988250 (+ 59us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:55.988300 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:55.988376 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:55.988520 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":256,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":63,"raft.queue_time_us":29477,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"thread_start_us":24866,"threads_started":1}
W20260812 19:13:56.018270 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 62378.9us
I20260812 19:13:56.071782 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:56.074838 24606 consensus_meta.cc:287] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.164s user 0.013s sys 0.007s
I20260812 19:13:56.081735 21288 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
W20260812 19:13:56.080842 21150 rpcz_store.cc:269] 0812 19:13:54.200248 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.203323 (+ 3075us) service_pool.cc:227] Handling call
0812 19:13:56.020457 (+1817134us) spinlock_profiling.cc:243] Waited 472 ms on lock 0x7b5400326254. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:56.020658 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1086237568}
I20260812 19:13:56.029738 24616 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:56.095500 24616 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:56.079875 24657 raft_consensus.cc:3037] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:56.087620 24506 raft_consensus.cc:686] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:56.110860 24513 consensus_queue.cc:1035] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:13:56.037947 21036 catalog_manager.cc:5495] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 4 to 6, leader changed from f741102bfe1441d1b453e5a838649059 (127.19.70.129) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 6 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260812 19:13:56.094581 24683 consensus_meta.cc:287] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.357s user 0.011s sys 0.010s
I20260812 19:13:56.112532 24456 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:56.122299 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:56.104111 24720 raft_consensus.cc:2781] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:13:56.119387 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:56.088153 21328 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:13:56.134686 21328 raft_consensus.cc:3037] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 1 FOLLOWER]: Advancing to term 2
W20260812 19:13:56.104069 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 77906.1us
I20260812 19:13:56.102597 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:56.111163 24453 consensus_queue.cc:260] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:56.103519 24626 raft_consensus.cc:2781] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:13:56.104907 24528 consensus_queue.cc:1046] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.004s
I20260812 19:13:56.113096 24506 consensus_queue.cc:237] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.143219 22008 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:56.145088 24298 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.146432 21152 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 8: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:56.069084 24649 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.081465 21329 rpcz_store.cc:276] 0812 19:13:52.804338 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.604101 (+1799763us) service_pool.cc:227] Handling call
0812 19:13:54.604519 (+ 418us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:54.604996 (+ 477us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:54.605037 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:54.639508 (+ 34471us) log.cc:842] Serialized 26 byte log entry
0812 19:13:55.549752 (+910244us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:55.549869 (+ 117us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:55.550056 (+ 187us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:55.912831 (+362775us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:13:56.021705 (+108874us) raft_consensus.cc:1671] finished
0812 19:13:56.021733 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:56.021985 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17928,"raft.queue_time_us":4259,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":109,"spinlock_wait_cycles":34013056,"thread_start_us":852093,"threads_started":1,"wal-append.queue_time_us":353818}
I20260812 19:13:56.147815 24683 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:56.153270 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:56.155205 24683 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:56.157485 21151 raft_consensus.cc:2379] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 4: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:56.158934 24720 raft_consensus.cc:686] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:56.160462 24720 consensus_queue.cc:237] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.161358 24656 raft_consensus.cc:3032] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:56.161830 24656 raft_consensus.cc:729] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:56.163123 21327 raft_consensus.cc:2379] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 6: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:56.164038 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:56.164163 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:56.164866 21156 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:13:56.165156 21157 raft_consensus.cc:2379] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 7: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:56.166064 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:56.166263 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:56.167410 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:56.149189 24626 raft_consensus.cc:686] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 6 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:56.164316 21200 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:56.136444 24559 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:56.100176 24553 consensus_queue.cc:1035] T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260812 19:13:56.183611 21156 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:56.184531 21328 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.189833 21155 raft_consensus.cc:1226] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:56.191263 21152 raft_consensus.cc:1226] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:56.192992 24453 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:56.193732 21149 raft_consensus.cc:2454] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:56.196133 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:56.197111 24528 raft_consensus.cc:2781] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 19:13:56.197561 21158 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:13:56.175980 24456 consensus_queue.cc:1046] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260812 19:13:56.139636 24694 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:56.207413 21157 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:56.205717 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:56.208876 21328 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:56.212172 21156 raft_consensus.cc:1226] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
W20260812 19:13:56.206840 24570 consensus_meta.cc:287] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.191s user 0.012s sys 0.008s
I20260812 19:13:56.217603 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:56.217625 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:56.220809 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:56.222194 24088 consensus_queue.cc:1035] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:13:56.226822 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:56.229884 21152 raft_consensus.cc:2379] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 6: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:56.197763 24528 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:56.234457 21235 raft_consensus.cc:2354] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 7. Current term is 9.
I20260812 19:13:56.235109 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50042 (request call id 369) took 2264 ms. Trace:
W20260812 19:13:56.235427 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 85096.3us
I20260812 19:13:56.236604 24649 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:56.235539 24574 raft_consensus.cc:2726] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:56.239269 24694 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:56.240272 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 52236.4us
I20260812 19:13:56.199783 24626 consensus_queue.cc:237] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.240309 21323 rpcz_store.cc:276] 0812 19:13:53.970283 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.898559 (+928276us) service_pool.cc:227] Handling call
0812 19:13:54.899183 (+ 624us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:55.553435 (+654252us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:55.553499 (+ 64us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:55.554752 (+ 1253us) log.cc:842] Serialized 26 byte log entry
0812 19:13:55.622599 (+ 67847us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:55.622747 (+ 148us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:55.638251 (+ 15504us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:56.230037 (+591786us) raft_consensus.cc:1671] finished
0812 19:13:56.230058 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:56.234903 (+ 4845us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":653417,"raft.queue_time_us":688769,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":293,"thread_start_us":66074,"threads_started":1,"wal-append.queue_time_us":259489}
I20260812 19:13:56.263630 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:56.265986 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 141528us
I20260812 19:13:56.245590 24606 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:56.267204 21036 catalog_manager.cc:5495] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:13:56.271852 21328 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:13:56.242856 24528 raft_consensus.cc:3037] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:56.249377 24298 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:56.261320 24616 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:56.277683 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:56.281365 21151 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:56.279901 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:56.190763 24656 consensus_queue.cc:260] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.275287 24576 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:56.279644 24559 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:56.301926 24751 log.cc:924] Time spent T aefbee13b3b74848be07bfcab278a3c8 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20260812 19:13:56.301048 21036 catalog_manager.cc:5495] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:13:56.306701 24674 consensus_queue.cc:1035] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:56.310508 24711 raft_consensus.cc:3032] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:13:56.310340 24576 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:56.311070 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:56.311883 21327 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:56.313027 24576 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:56.313796 22008 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:56.315603 24088 consensus_queue.cc:1046] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:13:56.315850 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:56.316696 24088 raft_consensus.cc:3032] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:56.317128 24088 raft_consensus.cc:729] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:56.281461 24500 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:13:56.321369 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 58815.7us
W20260812 19:13:56.323975 24750 log.cc:924] Time spent T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.171s user 0.000s sys 0.002s
I20260812 19:13:56.276859 24606 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:56.327689 21036 catalog_manager.cc:5495] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:13:56.344357 24574 consensus_queue.cc:1035] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260812 19:13:56.343891 21036 catalog_manager.cc:5495] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 1 to 5, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132). New cstate: current_term: 5 leader_uuid: "63db10063a7d4eb2a9cc9dc952654706" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:13:56.344765 24616 consensus_queue.cc:1046] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 19:13:56.345989 24616 consensus_queue.cc:1046] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 19:13:56.346815 24574 raft_consensus.cc:3032] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:56.346976 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:56.347276 24683 raft_consensus.cc:3032] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:56.347473 24664 raft_consensus.cc:3032] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:56.347764 24683 raft_consensus.cc:729] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:56.348012 24664 raft_consensus.cc:729] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:56.347296 24574 raft_consensus.cc:729] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:13:56.349217 21034 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48434 (request call id 251) took 14599 ms (client timeout 15000 ms). Trace:
I20260812 19:13:56.348992 24683 consensus_queue.cc:260] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:56.349352 24664 consensus_queue.cc:260] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:56.350049 24574 consensus_queue.cc:260] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:56.311115 24711 raft_consensus.cc:729] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:56.331970 24559 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.320272 24612 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:13:56.302668 24568 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:56.334425 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.335391 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:56.338927 24500 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.361428 21158 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8.
W20260812 19:13:56.362913 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 43995.6us
I20260812 19:13:56.371145 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.376124 21237 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:13:56.376037 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.377353 21239 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:56.373261 21243 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:56.370518 21112 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:56.382092 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.350559 24664 raft_consensus.cc:3037] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:56.330117 24656 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:56.382154 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.350589 24683 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:56.353473 24606 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:56.368110 24576 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:56.367612 24729 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:56.353150 24693 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:56.399909 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 81316.2us
I20260812 19:13:56.400440 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:56.351624 24574 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:56.387586 21238 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:56.355789 24492 consensus_queue.cc:1046] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260812 19:13:56.305387 24506 consensus_queue.cc:1046] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
I20260812 19:13:56.393468 21240 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:56.403538 24292 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:56.404063 24618 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:56.404526 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:56.408604 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:56.392879 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:56.407011 24626 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:56.416553 21243 raft_consensus.cc:1261] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260812 19:13:56.423727 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:56.410445 21036 catalog_manager.cc:5495] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 2 to 6, leader changed from 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 6 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:13:56.427960 21036 catalog_manager.cc:5495] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 2 to 5, leader changed from 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 5 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:13:56.434813 22053 raft_consensus.cc:1226] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:56.437162 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:56.424212 24638 raft_consensus.cc:2781] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:56.445361 21327 raft_consensus.cc:2379] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 8: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:56.447016 21240 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:56.409816 24693 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.453028 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.454452 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.430877 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:56.457540 22051 pending_rounds.cc:76] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706: Aborting all ops after (but not including) 6
I20260812 19:13:56.457968 22051 pending_rounds.cc:97] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260812 19:13:56.437996 24292 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:56.458316 22051 raft_consensus.cc:2866] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:56.459168 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:56.448776 24618 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:56.412441 24729 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.460894 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:56.457820 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:56.412588 24568 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:56.466840 21112 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:56.349720 21034 rpcz_store.cc:269] 0812 19:13:41.749487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:41.749828 (+ 341us) service_pool.cc:227] Handling call
0812 19:13:56.348948 (+14599120us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0812 19:13:47.207469 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0812 19:13:47.517038 (+309569us) write_op.cc:432] Acquiring schema lock in shared mode
0812 19:13:47.517122 (+ 84us) write_op.cc:435] Acquired schema lock
0812 19:13:47.517162 (+ 40us) tablet.cc:660] Decoding operations
0812 19:13:47.785639 (+268477us) write_op.cc:620] Acquiring the partition lock for write op
0812 19:13:47.785793 (+ 154us) write_op.cc:641] Partition lock acquired for write op
0812 19:13:47.785841 (+ 48us) tablet.cc:683] Acquiring locks for 25 operations
0812 19:13:47.830929 (+ 45088us) tablet.cc:699] Row locks acquired
0812 19:13:47.830981 (+ 52us) write_op.cc:260] PREPARE: finished
0812 19:13:47.831261 (+ 280us) write_op.cc:270] Start()
0812 19:13:47.831490 (+ 229us) write_op.cc:276] Timestamp: P: 1786562027831147 usec, L: 0
0812 19:13:47.831551 (+ 61us) op_driver.cc:331] REPLICATION: starting
0812 19:13:47.978812 (+147261us) log.cc:842] Serialized 9868 byte log entry
0812 19:13:49.372644 (+1393832us) op_driver.cc:444] REPLICATION: finished
0812 19:13:49.737531 (+364887us) write_op.cc:301] APPLY: starting
0812 19:13:49.745969 (+ 8438us) tablet.cc:1365] starting BulkCheckPresence
0812 19:13:49.757341 (+ 11372us) tablet.cc:1368] finished BulkCheckPresence
0812 19:13:49.757470 (+ 129us) tablet.cc:1370] starting ApplyRowOperation cycle
0812 19:13:49.870361 (+112891us) tablet.cc:1381] finished ApplyRowOperation cycle
0812 19:13:49.870926 (+ 565us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=25
0812 19:13:49.870972 (+ 46us) write_op.cc:312] APPLY: finished
0812 19:13:49.871524 (+ 552us) log.cc:842] Serialized 170 byte log entry
0812 19:13:50.124813 (+253289us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:13:50.151490 (+ 26677us) write_op.cc:454] Released schema lock
0812 19:13:50.200626 (+ 49136us) write_op.cc:341] FINISH: Updating metrics
0812 19:13:50.201044 (+ 418us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:13:50.201079 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":12706,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":332692,"apply.run_cpu_time_us":53550,"apply.run_wall_time_us":482885,"num_ops":25,"prepare.queue_time_us":465617,"prepare.run_cpu_time_us":47711,"prepare.run_wall_time_us":1183450,"raft.queue_time_us":978359,"raft.run_cpu_time_us":9292,"raft.run_wall_time_us":250462,"replication_time_us":1540959,"thread_start_us":1021697,"threads_started":5,"wal-append.queue_time_us":759860,"wal-append.run_cpu_time_us":90885,"wal-append.run_wall_time_us":2431901}]]}
W20260812 19:13:56.427304 21287 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 1.525s (SENT) [suppressed 21 similar messages]
I20260812 19:13:56.472461 21320 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:56.474258 21153 raft_consensus.cc:2454] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:56.450776 24492 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:56.436311 24458 consensus_queue.cc:1046] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260812 19:13:56.472582 21036 catalog_manager.cc:5495] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 2 to 5, leader changed from 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 5 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:13:56.484265 21243 raft_consensus.cc:1261] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260812 19:13:56.460242 24292 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:56.443899 24576 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:56.484092 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:56.483668 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:56.526810 21154 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 4.
I20260812 19:13:56.527056 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:56.527956 21320 raft_consensus.cc:2454] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 2.
I20260812 19:13:56.528658 24618 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:56.529840 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:56.530058 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:56.476653 24456 raft_consensus.cc:2781] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:13:56.523954 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:56.531760 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:56.535010 21155 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 8.
I20260812 19:13:56.535501 21323 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:56.534922 21036 catalog_manager.cc:5495] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:13:56.523550 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:56.445436 24626 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.528973 24568 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:56.533272 21327 raft_consensus.cc:1226] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:56.470145 24638 raft_consensus.cc:686] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:56.546198 21157 raft_consensus.cc:1226] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 7. Current term is 8. Ops: []
I20260812 19:13:56.484761 24500 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:56.556643 21329 raft_consensus.cc:2396] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260812 19:13:56.551457 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:56.483668 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:56.557135 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.572101 21243 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:13:56.431420 24088 consensus_queue.cc:260] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:56.543509 24758 log.cc:924] Time spent T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.145s user 0.001s sys 0.000s
I20260812 19:13:56.575253 21155 raft_consensus.cc:1226] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:56.534281 24559 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:56.534118 24456 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:56.587742 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:56.597244 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 47956.9us
I20260812 19:13:56.597607 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:56.603454 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:56.611960 24681 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:56.612720 21157 raft_consensus.cc:1261] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260812 19:13:56.603330 21036 catalog_manager.cc:5495] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: term changed from 4 to 6, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 6 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:13:56.615414 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.616513 21243 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:56.591220 24456 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:13:56.624708 21240 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:13:56.619181 24531 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:56.635677 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:56.627843 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.628268 21236 raft_consensus.cc:1261] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260812 19:13:56.628402 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 48479.4us
I20260812 19:13:56.628686 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:56.646813 21156 raft_consensus.cc:3032] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 19:13:56.629113 24657 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.612761 24681 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:56.631533 21155 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:56.632287 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:56.634589 24576 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:56.551188 24492 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.630264 21112 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:56.637332 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.623466 24710 raft_consensus.cc:3032] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:13:56.662460 21238 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:13:56.647245 21156 raft_consensus.cc:729] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:56.652940 21235 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:56.658572 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:56.665568 21157 raft_consensus.cc:2379] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:56.635196 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:56.657481 21036 catalog_manager.cc:5495] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 reported cstate change: config changed from index -1 to 2, term changed from 1 to 3, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:13:56.648205 22055 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:56.660199 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:56.672978 21240 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:56.668159 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.668517 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:56.674860 24754 log.cc:924] Time spent T aefbee13b3b74848be07bfcab278a3c8 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.223s user 0.002s sys 0.000s
I20260812 19:13:56.674255 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.678066 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:56.683228 21154 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1658) took 2486 ms (client timeout 2050 ms). Trace:
W20260812 19:13:56.683869 21154 rpcz_store.cc:269] 0812 19:13:54.185756 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:54.185920 (+ 164us) service_pool.cc:227] Handling call
0812 19:13:56.644101 (+2458181us) spinlock_profiling.cc:243] Waited 2.17 s on lock 0x7b54001239d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:56.671597 (+ 27496us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4995898112}
I20260812 19:13:56.663543 24710 raft_consensus.cc:729] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:56.685849 22053 raft_consensus.cc:1261] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260812 19:13:56.686867 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.687482 24492 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:56.687638 21233 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 4.
I20260812 19:13:56.687016 24710 consensus_queue.cc:260] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.679078 24638 consensus_queue.cc:237] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:56.691398 21156 consensus_queue.cc:260] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.692517 21156 raft_consensus.cc:3037] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 1 FOLLOWER]: Advancing to term 2
I20260812 19:13:56.641733 24088 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:56.706293 21112 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:56.709613 21203 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:56.725145 21151 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:13:56.726214 21158 raft_consensus.cc:1226] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 5. Current term is 6. Ops: []
W20260812 19:13:56.732766 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 305471us
I20260812 19:13:56.734338 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.735493 21233 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:13:56.733242 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:56.723229 21112 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:56.688925 24710 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:56.740711 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:56.749155 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.749480 21200 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:56.750021 21235 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:13:56.694151 24674 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 19:13:56.750842 21322 raft_consensus.cc:1226] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 3. Current term is 4. Ops: []
I20260812 19:13:56.754966 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:56.752436 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 82858us
I20260812 19:13:56.644068 24531 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.764047 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.765973 21243 raft_consensus.cc:2454] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 3.
I20260812 19:13:56.718924 24513 raft_consensus.cc:2781] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:56.768784 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:56.775305 21149 raft_consensus.cc:3032] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:56.752408 24458 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260812 19:13:56.757483 24629 raft_consensus.cc:2741] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260812 19:13:56.751881 24729 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:56.794423 21240 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:13:56.775957 21149 raft_consensus.cc:729] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:56.796155 24744 raft_consensus.cc:2726] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:56.796360 21149 consensus_queue.cc:260] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.797204 24531 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:56.799717 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 50319.7us
I20260812 19:13:56.784639 21322 raft_consensus.cc:1226] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:56.769052 24513 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:56.785327 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.752920 24657 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:56.758143 24674 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:56.801062 24759 log.cc:924] Time spent T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.268s user 0.001s sys 0.000s
I20260812 19:13:56.829421 24674 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:13:56.783774 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:56.806793 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:56.807085 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:56.765244 24576 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:56.809442 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:56.801496 21203 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:56.813851 21155 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:13:56.849871 21236 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:56.815119 24513 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:56.815984 21200 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:56.822306 21156 raft_consensus.cc:2454] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
I20260812 19:13:56.822906 24626 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:56.787218 24492 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:56.833315 24711 consensus_queue.cc:260] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.801898 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:56.827106 24693 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:56.859483 21200 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:56.889938 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:56.891881 21200 leader_election.cc:302] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:56.862385 21287 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:56.864238 21235 raft_consensus.cc:1261] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260812 19:13:56.866819 24576 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.868072 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:56.869405 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 57057.9us
I20260812 19:13:56.873636 24641 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:13:56.874778 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:56.907982 21328 raft_consensus.cc:2454] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:56.909683 21112 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:56.877525 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 76108.9us
I20260812 19:13:56.878283 24629 raft_consensus.cc:2781] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260812 19:13:56.881829 24528 consensus_meta.cc:287] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.589s user 0.009s sys 0.005s
I20260812 19:13:56.883441 21324 raft_consensus.cc:1261] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260812 19:13:56.912117 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:56.914378 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:56.941350 24576 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:56.941648 24574 consensus_meta.cc:287] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.532s user 0.008s sys 0.004s
I20260812 19:13:56.918737 24744 consensus_queue.cc:1046] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260812 19:13:56.943205 24744 raft_consensus.cc:3032] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:56.943608 24744 raft_consensus.cc:729] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:56.944499 24744 consensus_queue.cc:260] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:56.945868 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.946650 21235 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:56.947937 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:56.904646 24531 raft_consensus.cc:2781] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:56.956248 21154 raft_consensus.cc:1261] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260812 19:13:56.904369 24606 raft_consensus.cc:2781] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:56.915943 24629 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:56.953491 21200 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:56.964138 22056 raft_consensus.cc:2454] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:13:56.968966 21288 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:13:56.934573 24693 raft_consensus.cc:2781] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:56.933305 24492 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.974493 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:56.975556 21244 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:13:56.980219 24744 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:56.981153 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:56.960472 24711 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:56.982012 21240 raft_consensus.cc:2379] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 4: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:56.960841 24531 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:56.964812 24629 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:56.965077 24606 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:56.984913 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 131928us
I20260812 19:13:56.959434 24456 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:56.989218 21236 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:13:56.993695 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:56.977893 24693 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:56.978786 24626 raft_consensus.cc:2781] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:56.920190 24528 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:56.964128 24418 consensus_queue.cc:1035] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260812 19:13:57.032068 22053 raft_consensus.cc:1226] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:56.987764 24531 raft_consensus.cc:3037] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:57.033720 21149 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:57.049585 21152 raft_consensus.cc:2354] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 7. Current term is 8.
I20260812 19:13:57.057606 22053 raft_consensus.cc:1261] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260812 19:13:57.059026 22011 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:57.015105 24606 raft_consensus.cc:3037] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:56.987874 24500 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:57.019186 24693 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:57.020175 24626 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:57.029757 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:57.057107 24528 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:57.089823 24626 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:57.072211 24506 consensus_queue.cc:1046] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260812 19:13:57.115108 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:57.122514 24506 raft_consensus.cc:3032] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:57.123219 24506 raft_consensus.cc:729] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:57.124432 24506 consensus_queue.cc:260] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:57.096321 22047 raft_consensus.cc:2379] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 2: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:13:57.058045 24612 consensus_queue.cc:1046] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
W20260812 19:13:57.180305 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 258979us
I20260812 19:13:57.141810 24492 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:57.204762 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 99635.9us
I20260812 19:13:57.126910 24678 raft_consensus.cc:2726] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 19:13:57.156101 24664 consensus_meta.cc:287] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.763s user 0.011s sys 0.004s
I20260812 19:13:57.209162 21203 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:57.127090 24662 raft_consensus.cc:2781] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:13:57.210142 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:57.219966 21240 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:57.090713 24500 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:57.126575 24506 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:13:57.229262 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 252988us
I20260812 19:13:57.087874 24576 consensus_queue.cc:1035] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.122s
I20260812 19:13:57.255228 21149 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
W20260812 19:13:57.234668 24638 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.013s sys 0.027s
I20260812 19:13:57.224457 24662 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:57.283983 21327 raft_consensus.cc:1261] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260812 19:13:57.282526 21239 raft_consensus.cc:2454] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
W20260812 19:13:57.267786 24638 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s user 0.014s sys 0.028s
I20260812 19:13:57.250752 24664 raft_consensus.cc:2726] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:57.295714 21244 raft_consensus.cc:1261] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
W20260812 19:13:57.267253 24698 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260812 19:13:57.313151 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:57.333150 21319 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 8
I20260812 19:13:57.301438 21244 pending_rounds.cc:76] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa: Aborting all ops after (but not including) 5
I20260812 19:13:57.333803 21244 pending_rounds.cc:97] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260812 19:13:57.334160 21244 raft_consensus.cc:2866] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:13:57.336658 22047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40614 (request call id 513) took 1119 ms. Trace:
I20260812 19:13:57.337260 22047 rpcz_store.cc:276] 0812 19:13:56.216973 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:56.217222 (+ 249us) service_pool.cc:227] Handling call
0812 19:13:57.336309 (+1119087us) spinlock_profiling.cc:243] Waited 838 ms on lock 0x7b54006a0ed4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:57.336542 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1927440768}
I20260812 19:13:57.339566 22055 raft_consensus.cc:1226] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 1. Current term is 2. Ops: []
I20260812 19:13:57.345181 24492 raft_consensus.cc:3032] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:13:57.345144 24297 consensus_queue.cc:1035] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:13:57.345784 24492 raft_consensus.cc:729] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:57.284359 24662 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:13:57.353345 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 54367.2us
W20260812 19:13:57.300343 24088 consensus_meta.cc:287] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.579s user 0.005s sys 0.009s
W20260812 19:13:57.364354 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 275678us
I20260812 19:13:57.373535 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:57.374495 21240 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
W20260812 19:13:57.375967 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 107569us
I20260812 19:13:57.291772 24456 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:57.422947 24765 log.cc:924] Time spent T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.190s user 0.001s sys 0.000s
I20260812 19:13:57.292759 24513 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:57.427686 21150 raft_consensus.cc:2396] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260812 19:13:57.440815 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 47006.2us
I20260812 19:13:57.458537 21114 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:57.488366 21287 leader_election.cc:398] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4.
W20260812 19:13:57.500747 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 303765us
I20260812 19:13:57.317114 24778 consensus_queue.cc:1046] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.187s
I20260812 19:13:57.534812 21203 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:57.500679 24758 log.cc:924] Time spent T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.071s user 0.000s sys 0.001s
W20260812 19:13:57.539177 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 129105us
I20260812 19:13:57.544081 21240 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:13:57.346814 24492 consensus_queue.cc:260] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:57.493844 24694 raft_consensus.cc:2726] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260812 19:13:57.555142 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 176129us
I20260812 19:13:57.459223 24612 consensus_queue.cc:1035] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.070s
I20260812 19:13:57.510846 24664 raft_consensus.cc:2726] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:57.547636 21241 raft_consensus.cc:2932] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
W20260812 19:13:57.576067 24767 log.cc:924] Time spent T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.239s user 0.000s sys 0.000s
I20260812 19:13:57.594329 21203 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:57.557850 24297 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:57.609920 24778 raft_consensus.cc:2781] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:57.650240 24778 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:57.650665 24778 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:57.619952 24783 raft_consensus.cc:3032] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:57.652119 24783 raft_consensus.cc:729] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:57.652707 24788 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 19:13:57.653293 24788 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:57.653064 24783 consensus_queue.cc:260] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:57.654228 24710 consensus_meta.cc:287] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.899s user 0.015s sys 0.000s
W20260812 19:13:57.620227 24764 log.cc:924] Time spent T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.529s user 0.000s sys 0.002s
I20260812 19:13:57.656589 24292 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:57.656826 24500 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:57.625128 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 54043.8us
I20260812 19:13:57.657262 24292 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:57.658015 21114 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:57.653810 24788 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:13:57.659811 24292 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:57.662392 24774 log.cc:924] Time spent T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.179s user 0.000s sys 0.000s
I20260812 19:13:57.662868 24783 raft_consensus.cc:3037] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:57.663995 22048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 572) took 1035 ms. Trace:
I20260812 19:13:57.664443 22048 rpcz_store.cc:276] 0812 19:13:56.628423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:56.679329 (+ 50906us) service_pool.cc:227] Handling call
0812 19:13:56.680109 (+ 780us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:13:56.680817 (+ 708us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:56.680845 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:13:56.807555 (+126710us) log.cc:842] Serialized 590 byte log entry
0812 19:13:57.208615 (+401060us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:57.208679 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:57.208756 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:57.663720 (+454964us) raft_consensus.cc:1671] finished
0812 19:13:57.663740 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:57.663926 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7374,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"spinlock_wait_cycles":86016,"thread_start_us":389562,"threads_started":1,"wal-append.queue_time_us":502852}
I20260812 19:13:57.625916 24492 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Advancing to term 7
W20260812 19:13:57.667428 24641 consensus_meta.cc:287] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.747s user 0.013s sys 0.001s
I20260812 19:13:57.667958 24641 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260812 19:13:57.669318 24629 consensus_meta.cc:287] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.651s user 0.007s sys 0.007s
I20260812 19:13:57.669824 24629 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:57.673964 21150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1672) took 1191 ms. Trace:
I20260812 19:13:57.675717 21150 rpcz_store.cc:276] 0812 19:13:56.482656 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:56.482827 (+ 171us) service_pool.cc:227] Handling call
0812 19:13:57.673704 (+1190877us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b5400215054. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:57.673880 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2021965568}
I20260812 19:13:57.621435 21203 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:13:57.677862 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 56563.1us
I20260812 19:13:57.678396 21319 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:13:57.684005 21287 leader_election.cc:398] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4.
I20260812 19:13:57.648085 22051 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 569) took 1024 ms. Trace:
W20260812 19:13:57.605309 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 206344us
I20260812 19:13:57.694579 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:57.694972 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:57.697486 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:57.698458 21319 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:13:57.697734 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 254856us
I20260812 19:13:57.698890 21112 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:57.719209 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 61324.9us
I20260812 19:13:57.721302 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:57.722337 21320 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:13:57.700044 21241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1686) took 1327 ms. Trace:
I20260812 19:13:57.689280 24574 raft_consensus.cc:2726] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260812 19:13:57.695670 21152 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:13:57.689679 22051 rpcz_store.cc:276] 0812 19:13:56.624008 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:56.679282 (+ 55274us) service_pool.cc:227] Handling call
0812 19:13:56.680100 (+ 818us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:56.680755 (+ 655us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:56.680787 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:56.682138 (+ 1351us) log.cc:842] Serialized 564 byte log entry
0812 19:13:56.958297 (+276159us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:56.958340 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:56.958424 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:57.647773 (+689349us) raft_consensus.cc:1671] finished
0812 19:13:57.647801 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:57.648002 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":43136,"thread_start_us":275702,"threads_started":1,"wal-append.queue_time_us":406079}
W20260812 19:13:57.678771 24693 consensus_meta.cc:287] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.584s user 0.006s sys 0.005s
W20260812 19:13:57.705022 22008 leader_election.cc:334] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.599s (SENT)
I20260812 19:13:57.637959 24793 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:57.677011 24513 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:57.713506 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:57.645992 24506 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:57.733527 21241 rpcz_store.cc:276] 0812 19:13:56.373760 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:56.373944 (+ 184us) service_pool.cc:227] Handling call
0812 19:13:56.395412 (+ 21468us) raft_consensus.cc:1457] Updating replica for 3 ops
0812 19:13:56.438550 (+ 43138us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:56.438574 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0812 19:13:56.521921 (+ 83347us) log.cc:842] Serialized 616 byte log entry
0812 19:13:56.741467 (+219546us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:57.218339 (+476872us) log.cc:842] Serialized 14 byte log entry
0812 19:13:57.690856 (+472517us) log.cc:842] Serialized 14 byte log entry
0812 19:13:57.692735 (+ 1879us) log.cc:842] Serialized 14 byte log entry
0812 19:13:57.699249 (+ 6514us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:57.699354 (+ 105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:57.699410 (+ 56us) raft_consensus.cc:1671] finished
0812 19:13:57.699420 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:57.699967 (+ 547us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16381,"prepare.queue_time_us":341031,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":53,"raft.queue_time_us":282530,"raft.run_cpu_time_us":849,"raft.run_wall_time_us":54336,"spinlock_wait_cycles":24231168,"thread_start_us":931252,"threads_started":3,"wal-append.queue_time_us":456376}
I20260812 19:13:57.746452 21112 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:57.780299 21150 raft_consensus.cc:2379] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 2: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:57.750761 22008 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:57.751344 24793 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:57.752360 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:57.753260 24629 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:57.695396 24681 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:57.755930 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 2016) took 1115 ms. Trace:
I20260812 19:13:57.786933 21114 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:57.788057 24683 raft_consensus.cc:2781] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:57.788561 24683 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:57.788971 24683 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:57.790202 24793 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:57.710471 24456 raft_consensus.cc:2781] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:57.793092 24456 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:57.793550 24456 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
W20260812 19:13:57.794478 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 94067.8us
W20260812 19:13:57.796658 24531 consensus_meta.cc:287] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.705s user 0.012s sys 0.000s
I20260812 19:13:57.797168 24531 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:57.797916 24807 raft_consensus.cc:2726] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 19:13:57.760015 24744 consensus_meta.cc:287] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.773s user 0.014s sys 0.000s
I20260812 19:13:57.747460 24458 raft_consensus.cc:2781] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader election won for term 8
I20260812 19:13:57.802306 24683 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:57.803458 24458 raft_consensus.cc:686] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 8 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:57.803818 21150 raft_consensus.cc:1226] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:57.806277 24531 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:57.808655 24683 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:57.809688 24626 consensus_meta.cc:287] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.707s user 0.008s sys 0.005s
I20260812 19:13:57.810202 24626 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:57.789060 21114 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:57.770757 24693 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:57.643079 24297 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:57.773054 24711 consensus_meta.cc:287] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.786s user 0.014s sys 0.000s
I20260812 19:13:57.742898 24720 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:57.778697 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1769) took 1018 ms. Trace:
I20260812 19:13:57.820637 21329 rpcz_store.cc:276] 0812 19:13:56.760285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:56.774753 (+ 14468us) service_pool.cc:227] Handling call
0812 19:13:56.792548 (+ 17795us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:56.809423 (+ 16875us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:13:56.809449 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:56.861894 (+ 52445us) log.cc:842] Serialized 26 byte log entry
0812 19:13:57.183324 (+321430us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:57.183366 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:57.183452 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:57.778388 (+594936us) raft_consensus.cc:1671] finished
0812 19:13:57.778411 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:57.778593 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":264288,"threads_started":1,"wal-append.queue_time_us":628637}
I20260812 19:13:57.786801 21318 rpcz_store.cc:276] 0812 19:13:56.640827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:56.641028 (+ 201us) service_pool.cc:227] Handling call
0812 19:13:56.673097 (+ 32069us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:56.719462 (+ 46365us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:56.719496 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:56.754406 (+ 34910us) log.cc:842] Serialized 26 byte log entry
0812 19:13:57.214160 (+459754us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:13:57.214197 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:57.214265 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:13:57.755632 (+541367us) raft_consensus.cc:1671] finished
0812 19:13:57.755656 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:57.755857 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7663,"raft.queue_time_us":7099,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":33553,"thread_start_us":433113,"threads_started":1,"wal-append.queue_time_us":945014}
W20260812 19:13:57.820668 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 74332.1us
I20260812 19:13:57.822999 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:57.823657 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:57.823805 21154 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:13:57.762380 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 130388us
W20260812 19:13:57.756889 24674 consensus_meta.cc:287] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.917s user 0.004s sys 0.009s
I20260812 19:13:57.824374 21151 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:57.828696 22045 raft_consensus.cc:2396] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260812 19:13:57.828508 21152 raft_consensus.cc:1226] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
I20260812 19:13:57.829746 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:57.826284 24681 raft_consensus.cc:2781] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:57.841041 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:57.841238 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:57.842878 21153 raft_consensus.cc:3037] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:57.843889 21328 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:57.846122 21318 raft_consensus.cc:1226] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
I20260812 19:13:57.848948 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:57.849558 21320 raft_consensus.cc:3037] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 4
I20260812 19:13:57.857203 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:57.858208 21329 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 6
W20260812 19:13:57.866410 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 77429.7us
I20260812 19:13:57.867441 21114 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:57.869839 21319 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:57.826740 24794 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:57.872781 21154 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:57.838076 24681 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:57.875631 21200 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:57.810961 24458 consensus_queue.cc:237] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:57.842303 24553 raft_consensus.cc:2726] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:57.811216 24506 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:57.874919 21287 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:57.876612 21112 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:57.856025 24744 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:57.897799 21114 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:57.876051 24672 raft_consensus.cc:2726] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:57.832535 24720 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:57.882064 24681 raft_consensus.cc:3037] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:57.831826 24674 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:57.926463 21204 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
W20260812 19:13:57.951723 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 76184.6us
I20260812 19:13:57.881767 24794 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:57.929761 22046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 566) took 1364 ms. Trace:
W20260812 19:13:57.930751 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 87944.1us
I20260812 19:13:57.934579 24553 consensus_queue.cc:1035] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:13:57.938645 24693 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:57.939213 24781 log.cc:924] Time spent T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.134s user 0.000s sys 0.001s
I20260812 19:13:57.943447 24626 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:57.912211 24418 raft_consensus.cc:2781] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election won for term 6
W20260812 19:13:57.966559 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 221390us
I20260812 19:13:57.967842 21204 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:57.944903 24678 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:13:57.946975 24816 log.cc:924] Time spent T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1: Append to log took a long time: real 0.057s user 0.002s sys 0.000s
W20260812 19:13:57.915057 24606 consensus_meta.cc:287] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.824s user 0.015s sys 0.000s
I20260812 19:13:57.914525 24531 raft_consensus.cc:2781] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:57.976827 24785 raft_consensus.cc:2726] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:57.981541 24606 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:57.983964 24606 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:57.950977 24641 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:57.919883 24513 raft_consensus.cc:2781] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:57.918735 24568 raft_consensus.cc:2726] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:57.866900 24788 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:57.972227 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:57.887974 24662 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:57.905115 24744 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:58.005851 21153 raft_consensus.cc:2454] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
W20260812 19:13:58.009891 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 112216us
W20260812 19:13:58.012844 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 136342us
I20260812 19:13:58.015826 21114 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:58.018639 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:57.987110 24492 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:57.987198 24418 raft_consensus.cc:686] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:58.024606 21319 raft_consensus.cc:2379] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 5: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
W20260812 19:13:58.030738 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 167157us
I20260812 19:13:57.990098 24807 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:57.981608 24531 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:58.026086 22048 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:57.995456 24297 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:57.993171 24513 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:57.998348 24292 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:57.966933 24787 consensus_queue.cc:1046] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260812 19:13:58.026012 24693 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:58.045516 24531 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:58.055150 24418 consensus_queue.cc:237] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:58.055799 24788 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:58.009485 24783 consensus_queue.cc:1046] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260812 19:13:58.039919 24683 raft_consensus.cc:2781] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:13:58.087725 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:58.094422 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:57.997294 24641 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:58.011725 24456 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:58.034011 24606 raft_consensus.cc:2726] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:13:58.015724 24576 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.036s
W20260812 19:13:58.094803 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 103764us
I20260812 19:13:58.051994 24513 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
W20260812 19:13:58.106313 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 99623.1us
I20260812 19:13:58.152561 22045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40614 (request call id 522) took 1198 ms. Trace:
I20260812 19:13:58.037029 24492 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:58.148432 22052 raft_consensus.cc:2932] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Committing config change with OpId 4.4: config changed from index 2 to 4, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
W20260812 19:13:58.154896 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 98760.1us
I20260812 19:13:58.155059 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:58.156420 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:13:58.182111 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 144650us
I20260812 19:13:58.185958 21114 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:58.153030 22045 rpcz_store.cc:276] 0812 19:13:56.953686 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:56.976488 (+ 22802us) service_pool.cc:227] Handling call
0812 19:13:58.141756 (+1165268us) spinlock_profiling.cc:243] Waited 736 ms on lock 0x7b540061fb54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:58.152483 (+ 10727us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1693499648}
I20260812 19:13:58.085709 24292 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:58.183665 21327 raft_consensus.cc:3037] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:58.111642 24788 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:58.184393 21203 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:58.167682 21157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:54786 (request call id 392) took 1482 ms. Trace:
I20260812 19:13:58.141570 24528 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:58.195261 21203 leader_election.cc:302] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:58.193511 21157 rpcz_store.cc:276] 0812 19:13:56.669709 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:56.710468 (+ 40759us) service_pool.cc:227] Handling call
0812 19:13:58.144382 (+1433914us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400182754. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2baf0 00007f52d9aaef2a 00007f52de9ebc6e 00007f52d80ee807 00007f52d80ee792 00007f52d80ee6c1 00007f52d80ee63c 00007f52d80ed8f2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:58.144994 (+ 612us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2803572480}
I20260812 19:13:58.198125 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:58.201522 21114 leader_election.cc:302] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:57.998700 24678 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:58.210275 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:58.212262 21114 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:57.986057 24453 raft_consensus.cc:2932] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Committing config change with OpId 4.4: config changed from index 2 to 4, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:13:58.217207 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:58.218128 21150 raft_consensus.cc:2379] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 9: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:13:58.218859 24809 log.cc:924] Time spent T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
I20260812 19:13:58.219477 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:13:58.223837 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:58.225101 21149 raft_consensus.cc:2379] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 6: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:58.226763 24825 raft_consensus.cc:2741] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260812 19:13:58.227029 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:58.212323 24783 raft_consensus.cc:2781] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:13:58.228212 24783 raft_consensus.cc:686] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:58.229219 24783 consensus_queue.cc:237] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:58.229745 24812 consensus_queue.cc:1046] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260812 19:13:58.153482 24506 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:58.236145 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:58.237135 21319 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 9
I20260812 19:13:58.232769 21114 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:13:58.188472 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:58.240587 21114 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:58.227416 24819 raft_consensus.cc:2781] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:13:58.215044 21157 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:58.187671 24683 raft_consensus.cc:686] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:58.045089 24807 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:58.100687 24693 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:58.158706 24576 consensus_queue.cc:1225] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [LEADER]: Peer d1e1da0b9585460e8c365bf55679d4fa log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260812 19:13:58.209921 24674 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:58.146829 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:58.264822 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:58.261034 21203 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:58.232379 24297 raft_consensus.cc:2726] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:58.244060 24698 raft_consensus.cc:2726] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:58.237797 24720 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:58.280118 21149 raft_consensus.cc:2354] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
W20260812 19:13:58.281494 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 313750us
I20260812 19:13:58.199136 21321 raft_consensus.cc:2932] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Committing config change with OpId 4.4: config changed from index 2 to 4, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:13:58.203873 24788 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:58.208271 24528 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:58.329285 21318 raft_consensus.cc:1261] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260812 19:13:58.333981 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 136195us
I20260812 19:13:58.337581 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:58.338434 22051 raft_consensus.cc:2454] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 2.
W20260812 19:13:58.338747 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 77784us
I20260812 19:13:58.339609 21115 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:58.339669 21288 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
W20260812 19:13:58.340544 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 100072us
I20260812 19:13:58.341297 21114 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:13:58.342110 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:58.343123 21287 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:13:58.259298 24568 consensus_queue.cc:1046] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260812 19:13:58.347824 21319 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 9.
I20260812 19:13:58.301795 24292 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:58.356978 21114 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:58.343945 21204 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:58.302225 24088 raft_consensus.cc:2726] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:13:58.358176 21323 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:58.249543 24825 consensus_queue.cc:1035] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.090s
I20260812 19:13:58.273873 24819 raft_consensus.cc:686] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:58.327430 24744 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:58.368871 21320 consensus_meta.cc:287] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.519s user 0.013s sys 0.004s
I20260812 19:13:58.378782 21320 raft_consensus.cc:2454] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:58.303462 24662 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:58.281684 24794 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:58.347203 24297 raft_consensus.cc:2726] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:13:58.292192 22047 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:58.360626 21329 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
W20260812 19:13:58.397888 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 54038.8us
I20260812 19:13:58.399171 21204 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:13:58.402550 21204 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:58.366441 24720 raft_consensus.cc:2781] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:58.410120 22011 leader_election.cc:398] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:13:58.410663 22011 leader_election.cc:401] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260812 19:13:57.963853 22046 rpcz_store.cc:276] 0812 19:13:56.455177 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:56.522879 (+ 67702us) service_pool.cc:227] Handling call
0812 19:13:56.587455 (+ 64576us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:56.630869 (+ 43414us) raft_consensus.cc:1520] Early marking committed up to index 8
0812 19:13:57.065204 (+434335us) log.cc:842] Serialized 14 byte log entry
0812 19:13:57.741594 (+676390us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:57.743891 (+ 2297us) raft_consensus.cc:1631] Marking committed up to 8
0812 19:13:57.777807 (+ 33916us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:57.783337 (+ 5530us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:57.819591 (+ 36254us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17665,"prepare.queue_time_us":515388,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":77,"raft.queue_time_us":157583,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"thread_start_us":923189,"threads_started":3,"wal-append.queue_time_us":533269}
I20260812 19:13:58.412204 21204 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:58.363888 24698 raft_consensus.cc:2781] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:13:58.409863 24720 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:58.424110 21157 raft_consensus.cc:1226] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:58.419332 21115 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:58.496562 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:58.497038 24662 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:58.499881 24693 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:58.500161 24662 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:13:58.500856 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 157844us
I20260812 19:13:58.502156 21287 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:58.503589 24506 consensus_queue.cc:1046] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260812 19:13:58.497705 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 72586.7us
I20260812 19:13:58.500145 21154 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:58.389956 24574 raft_consensus.cc:2726] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:58.512825 24830 consensus_queue.cc:1035] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 19:13:58.503671 24662 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:58.499219 24847 raft_consensus.cc:2726] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:58.405519 24576 raft_consensus.cc:2781] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:58.402837 24456 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:58.402484 24641 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:58.417912 24626 raft_consensus.cc:2726] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:13:58.558815 22011 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:58.388970 24492 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:13:58.472273 24778 consensus_meta.cc:287] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.821s user 0.014s sys 0.000s
I20260812 19:13:58.424799 24458 raft_consensus.cc:2726] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260812 19:13:58.423703 24794 raft_consensus.cc:2726] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:58.514428 24576 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:58.421052 24720 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:58.418193 24834 raft_consensus.cc:2726] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:58.564677 24672 raft_consensus.cc:2781] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:58.472747 24698 raft_consensus.cc:686] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:13:58.511287 24844 raft_consensus.cc:2781] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader election won for term 8
I20260812 19:13:58.588057 21287 leader_election.cc:398] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 7. Current term is 9.
W20260812 19:13:58.593140 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 217399us
I20260812 19:13:58.603473 21287 leader_election.cc:401] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:13:58.389379 24683 consensus_queue.cc:237] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:58.581529 24531 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:58.583544 24662 raft_consensus.cc:2781] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:58.421674 24835 consensus_queue.cc:1035] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.163s
I20260812 19:13:58.581058 24513 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:58.596388 24576 raft_consensus.cc:3037] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:58.635485 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:58.652360 22048 consensus_meta.cc:287] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.612s user 0.014s sys 0.000s
I20260812 19:13:58.653017 22048 raft_consensus.cc:2454] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:13:58.422750 24819 consensus_queue.cc:237] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:58.616151 24662 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:58.665748 24531 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:58.666851 24788 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:58.654103 24626 raft_consensus.cc:2726] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 7. Current term is 9.
I20260812 19:13:58.635354 24844 raft_consensus.cc:686] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 8 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:58.639292 24418 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:58.679237 24662 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:58.697271 24418 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:58.698550 24839 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:58.699184 24839 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:58.699733 24418 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:58.701043 24418 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:58.701416 24839 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:58.701526 24418 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:58.677809 24297 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:58.703722 24678 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:58.703555 24297 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:58.704497 24807 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:58.705775 24297 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:58.705614 24844 consensus_queue.cc:237] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:13:58.680620 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 116721us
W20260812 19:13:58.714448 21327 consensus_meta.cc:287] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.517s user 0.011s sys 0.001s
I20260812 19:13:58.715231 21327 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:13:58.681383 24698 consensus_queue.cc:237] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:58.666396 24711 raft_consensus.cc:2726] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:13:58.637936 24528 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:58.587591 24778 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:58.656188 24693 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:58.720364 21329 raft_consensus.cc:1261] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260812 19:13:58.737293 21287 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:58.720259 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:58.732743 22052 consensus_meta.cc:287] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.547s user 0.012s sys 0.000s
I20260812 19:13:58.738368 21320 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:58.750149 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:58.757434 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 251696us
I20260812 19:13:58.760787 21200 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:58.761880 24629 raft_consensus.cc:2781] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:58.763017 24629 raft_consensus.cc:686] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:58.764307 24629 consensus_queue.cc:237] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:58.765215 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:58.777074 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:58.775015 24678 raft_consensus.cc:2726] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:13:58.780980 24513 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:58.781724 24418 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:58.782130 24683 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:58.783499 21287 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:58.771317 24568 consensus_queue.cc:1035] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260812 19:13:58.785086 21323 raft_consensus.cc:2454] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
W20260812 19:13:58.771713 24656 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.006s sys 0.021s
I20260812 19:13:58.793300 21200 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:58.800918 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:58.801911 21329 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:58.805380 24840 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:58.806280 24840 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:58.810004 24840 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:58.821269 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 409134us
W20260812 19:13:58.797880 24656 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.704s user 0.007s sys 0.021s
I20260812 19:13:58.839469 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:58.833882 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:58.827397 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:58.827904 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:58.841121 21241 raft_consensus.cc:2454] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:13:58.841473 21157 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
W20260812 19:13:58.820727 24681 consensus_meta.cc:287] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.860s user 0.015s sys 0.000s
I20260812 19:13:58.876081 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:58.816687 24843 raft_consensus.cc:2781] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:58.888236 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:58.877172 21235 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 8.
I20260812 19:13:58.878741 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:58.880442 21240 raft_consensus.cc:1226] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 6. Current term is 7. Ops: []
I20260812 19:13:58.880966 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:58.882221 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:58.820427 24787 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:58.894022 21324 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
W20260812 19:13:58.898484 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 123005us
I20260812 19:13:58.900614 21329 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:13:58.901732 21287 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:58.901911 21200 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:58.843225 24531 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:58.783799 24693 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:58.891990 21244 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:58.916929 21203 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:58.782781 24683 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:58.921398 21287 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:58.925019 21243 raft_consensus.cc:1226] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5. Ops: []
I20260812 19:13:58.928412 24847 raft_consensus.cc:2726] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260812 19:13:58.928903 24453 consensus_meta.cc:287] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.554s user 0.009s sys 0.002s
W20260812 19:13:58.933074 21321 consensus_meta.cc:287] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.628s user 0.010s sys 0.000s
I20260812 19:13:58.935387 24837 raft_consensus.cc:3032] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:13:58.935951 24837 raft_consensus.cc:729] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:58.936926 24837 consensus_queue.cc:260] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:58.938711 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:58.937760 24683 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:58.938414 24808 log.cc:924] Time spent T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260812 19:13:58.884033 24843 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:58.940270 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:58.967818 22051 raft_consensus.cc:2454] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 2.
I20260812 19:13:58.940708 24678 raft_consensus.cc:2741] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260812 19:13:58.938493 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:58.924338 21287 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:58.874244 24820 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:58.969491 22047 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:58.884364 24854 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:58.881251 24836 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:13:58.975353 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 51107us
I20260812 19:13:58.977424 21154 raft_consensus.cc:2454] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:58.980829 21287 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:58.974349 21157 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:13:58.918653 24832 raft_consensus.cc:2781] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:58.923363 24830 raft_consensus.cc:2781] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:13:58.979417 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:58.991225 21240 raft_consensus.cc:1226] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:58.981676 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:58.923446 24787 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:58.940302 24813 raft_consensus.cc:3032] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:58.995031 24830 raft_consensus.cc:686] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:58.944746 24853 raft_consensus.cc:2781] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:58.942435 24531 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:58.978765 24844 raft_consensus.cc:2741] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260812 19:13:58.887854 24681 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:58.984241 24693 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:59.012658 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.013322 21240 raft_consensus.cc:1261] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260812 19:13:58.997460 24662 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:58.997726 24678 raft_consensus.cc:2781] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:13:58.973513 24843 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:59.009011 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:59.011016 24853 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:59.020484 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:58.979809 24820 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:58.998009 24813 raft_consensus.cc:729] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:58.991695 24832 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:58.987548 24836 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:59.026408 21243 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:58.989435 24854 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:59.031332 21200 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:59.020079 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:59.037369 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.019596 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:59.017475 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:59.054262 21236 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:59.015134 24787 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:59.054266 21150 raft_consensus.cc:2454] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:13:59.023610 21157 raft_consensus.cc:1226] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 5. Current term is 6. Ops: []
I20260812 19:13:59.017422 21155 raft_consensus.cc:2379] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:59.009044 21115 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:59.059033 22011 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:59.042014 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:58.980024 24807 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:59.048929 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.017007 24606 raft_consensus.cc:3032] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:13:59.063596 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.064431 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.069114 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.065733 21324 raft_consensus.cc:2354] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:13:59.019423 24778 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:59.066866 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 57912.3us
I20260812 19:13:59.017089 24681 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:59.020740 24576 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:59.068039 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:59.098860 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 282474us
I20260812 19:13:59.040431 24853 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:59.049330 24832 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:59.101032 21244 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
I20260812 19:13:59.049232 24531 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:59.045900 24678 raft_consensus.cc:686] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 6 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:59.108165 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.128516 21323 raft_consensus.cc:2454] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:13:59.013666 24830 consensus_queue.cc:237] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:59.128119 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:59.068118 24819 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader election won for term 9
I20260812 19:13:59.116247 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.069868 24088 raft_consensus.cc:2781] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 19:13:59.078580 24492 raft_consensus.cc:2781] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:59.068652 24606 raft_consensus.cc:729] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:59.130070 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:59.067096 24837 raft_consensus.cc:3037] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:13:59.130849 21236 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:59.132675 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:59.160198 21243 raft_consensus.cc:2454] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:13:59.107868 24787 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:59.136155 21322 raft_consensus.cc:2379] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 4: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:59.162971 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:59.166767 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:59.167232 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:59.168071 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:59.162324 24674 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.731s user 0.020s sys 0.052s
I20260812 19:13:59.135731 24088 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:59.180758 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:59.104512 24807 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:59.182304 22055 raft_consensus.cc:1261] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260812 19:13:59.149387 24492 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:59.157536 24819 raft_consensus.cc:686] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 9 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:13:59.177626 24787 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:59.192600 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.180382 24820 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:59.201174 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:59.187034 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:59.185963 24674 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.770s user 0.020s sys 0.053s
I20260812 19:13:59.180915 24794 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:59.217481 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1803) took 1077 ms. Trace:
I20260812 19:13:59.166312 24683 consensus_queue.cc:1035] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260812 19:13:59.227715 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.228192 24854 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:59.190260 24088 raft_consensus.cc:3037] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:59.204320 24785 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:59.230243 21241 raft_consensus.cc:1261] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260812 19:13:59.230914 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:59.231632 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:13:59.232275 24861 log.cc:924] Time spent T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.225s user 0.001s sys 0.000s
I20260812 19:13:59.225718 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:59.187856 24531 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:59.238904 21318 raft_consensus.cc:2379] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 3: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:59.193265 24492 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:59.240945 22052 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 591) took 1136 ms. Trace:
I20260812 19:13:59.241818 21243 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:59.133214 24813 consensus_queue.cc:260] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:59.260329 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:59.180203 24606 consensus_queue.cc:260] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:59.260931 22011 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:13:59.260509 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:59.259093 22052 rpcz_store.cc:276] 0812 19:13:58.104210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:58.104475 (+ 265us) service_pool.cc:227] Handling call
0812 19:13:58.147919 (+ 43444us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:58.148254 (+ 335us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:13:58.738427 (+590173us) log.cc:842] Serialized 14 byte log entry
0812 19:13:59.207910 (+469483us) log.cc:842] Serialized 14 byte log entry
0812 19:13:59.240387 (+ 32477us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:59.240472 (+ 85us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:59.240543 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:59.240660 (+ 117us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:59.240865 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36352,"raft.queue_time_us":36834,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":6906,"thread_start_us":441060,"threads_started":3,"wal-append.queue_time_us":454284}
I20260812 19:13:59.271761 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:59.272583 21233 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:59.273232 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:59.260967 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:59.274379 21240 raft_consensus.cc:2379] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:59.273933 21238 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:59.276976 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.282085 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:59.283054 21319 raft_consensus.cc:2354] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 2. Current term is 3.
I20260812 19:13:59.283270 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.285178 21114 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:59.286528 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.292033 21329 raft_consensus.cc:1203] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 5.5->[6.6-6.6] Dedup: 6.6->[]
I20260812 19:13:59.292754 21329 raft_consensus.cc:1226] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260812 19:13:59.299153 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:59.260339 24662 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:59.261009 24793 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:59.331612 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:13:59.332502 22052 raft_consensus.cc:3037] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 2 FOLLOWER]: Advancing to term 3
I20260812 19:13:59.306195 24858 raft_consensus.cc:2726] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260812 19:13:59.307507 22008 leader_election.cc:334] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.508s (SENT)
I20260812 19:13:59.308756 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:59.335027 22008 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:13:59.335203 21238 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:59.309620 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.310289 22050 raft_consensus.cc:1261] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260812 19:13:59.311038 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.311813 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.312582 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.314929 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:59.337189 21240 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:13:59.314849 21233 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:13:59.317747 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:13:59.304500 24863 log.cc:924] Time spent T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260812 19:13:59.235754 24794 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:59.334144 21114 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:59.343387 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:59.348220 21235 raft_consensus.cc:2362] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5. Re-sending same reply.
I20260812 19:13:59.344959 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:59.345070 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.349977 21327 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:13:59.253412 24785 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:59.265091 24528 consensus_queue.cc:1046] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20260812 19:13:59.351397 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.263783 24531 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:59.194104 24819 consensus_queue.cc:237] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:59.360987 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:59.347244 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.373971 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.385897 21319 raft_consensus.cc:2354] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:13:59.374658 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:59.297544 24681 consensus_queue.cc:1035] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.074s
I20260812 19:13:59.233913 24678 consensus_queue.cc:237] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:13:59.373503 21318 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
W20260812 19:13:59.381834 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 47802.9us
I20260812 19:13:59.404753 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.406939 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.389729 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:13:59.392879 21233 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:59.333999 24576 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:13:59.390161 21112 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:59.343150 24812 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:59.348316 24662 raft_consensus.cc:2781] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 19:13:59.376811 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:59.416035 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:59.416442 21328 raft_consensus.cc:2379] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:13:59.381984 24606 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:59.398283 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:59.417605 22052 raft_consensus.cc:2454] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:13:59.368487 24836 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:13:59.418028 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 100855us
I20260812 19:13:59.418846 21157 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 8.
I20260812 19:13:59.427228 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.427898 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:59.428222 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.427923 21203 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:59.395460 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.408028 21239 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:13:59.391626 24834 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:13:59.344542 24793 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:59.425554 21327 raft_consensus.cc:2354] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:13:59.429831 21319 raft_consensus.cc:2379] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 5: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:59.387177 24418 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:13:59.408056 24683 raft_consensus.cc:2726] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260812 19:13:59.427027 21288 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:13:59.427932 21329 raft_consensus.cc:2454] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:13:59.415395 24531 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:59.415745 24662 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:13:59.425972 24506 raft_consensus.cc:2726] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:59.424827 24453 raft_consensus.cc:2781] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:59.422092 24807 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:59.459930 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.449095 21112 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:59.466348 21324 raft_consensus.cc:1261] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260812 19:13:59.466269 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.473963 21238 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:13:59.457193 21328 raft_consensus.cc:1261] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260812 19:13:59.484550 21327 raft_consensus.cc:2379] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 4: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:13:59.435570 24830 raft_consensus.cc:2781] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 19:13:59.408142 24638 consensus_queue.cc:1035] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260812 19:13:59.470530 21200 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:13:59.486994 24870 log.cc:924] Time spent T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.184s user 0.001s sys 0.000s
I20260812 19:13:59.480145 22011 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:13:59.499740 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 50760.2us
I20260812 19:13:59.420921 24812 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:59.455842 24787 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:13:59.471204 24877 consensus_queue.cc:1046] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20260812 19:13:59.521323 21112 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:59.522951 22011 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:59.504195 24531 raft_consensus.cc:2781] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:13:59.441066 24418 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:59.480508 24568 raft_consensus.cc:2781] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 19:13:59.617400 24568 raft_consensus.cc:686] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 3 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:59.618466 24568 consensus_queue.cc:237] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:59.619640 21149 raft_consensus.cc:1261] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260812 19:13:59.528770 24681 raft_consensus.cc:2781] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:59.622761 24877 raft_consensus.cc:3032] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:13:59.622913 24681 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:59.623636 24681 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:59.627674 24877 raft_consensus.cc:729] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:13:59.629395 24877 consensus_queue.cc:260] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:13:59.631059 24877 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:13:59.532665 24720 consensus_meta.cc:287] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.920s user 0.014s sys 0.000s
I20260812 19:13:59.632831 24720 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:59.498939 24672 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:13:59.634909 24672 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:59.463110 24662 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:13:59.565713 21243 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:59.637604 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:59.638015 21243 raft_consensus.cc:1261] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260812 19:13:59.638528 24638 raft_consensus.cc:2781] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:59.639214 24638 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:59.639716 24638 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:59.603590 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:13:59.606259 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 84700.9us
I20260812 19:13:59.604444 24531 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:59.645279 24531 raft_consensus.cc:3037] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:13:59.647305 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.638306 21157 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:59.656545 24794 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:59.657823 22048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40614 (request call id 528) took 1713 ms. Trace:
I20260812 19:13:59.658119 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:59.471830 24453 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:59.658296 22048 rpcz_store.cc:276] 0812 19:13:57.944241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:57.958882 (+ 14641us) service_pool.cc:227] Handling call
0812 19:13:59.657610 (+1698728us) spinlock_profiling.cc:243] Waited 26 ms on lock 0x7b540049be54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:59.657756 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":59864960}
I20260812 19:13:59.658890 21149 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:59.658923 24453 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:59.663805 24720 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:59.667075 21324 raft_consensus.cc:1261] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260812 19:13:59.506129 24506 consensus_queue.cc:1035] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.112s
I20260812 19:13:59.500005 24830 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:13:59.613135 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 90253.9us
I20260812 19:13:59.614975 24807 raft_consensus.cc:2781] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:13:59.615698 24895 raft_consensus.cc:2781] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:13:59.616640 24793 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:59.622113 24641 consensus_queue.cc:1035] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:13:59.610111 21112 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
W20260812 19:13:59.667582 24867 log.cc:924] Time spent T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.157s user 0.000s sys 0.000s
I20260812 19:13:59.681972 21203 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:59.435698 24834 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:13:59.759472 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:59.697062 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:13:59.807292 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 197286us
W20260812 19:13:59.801458 21153 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51004 (request call id 390) took 1757 ms (client timeout 1853 ms). Trace:
I20260812 19:13:59.732930 21233 raft_consensus.cc:2454] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:13:59.802380 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:13:59.812489 21153 rpcz_store.cc:269] 0812 19:13:58.043890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:58.044177 (+ 287us) service_pool.cc:227] Handling call
0812 19:13:59.801119 (+1756942us) spinlock_profiling.cc:243] Waited 624 ms on lock 0x7b54001a3e54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:13:59.801362 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1435112832}
I20260812 19:13:59.813608 21112 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:13:59.815990 21157 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:13:59.762576 21238 raft_consensus.cc:1261] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260812 19:13:59.709657 21288 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
W20260812 19:13:59.849088 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 139589us
I20260812 19:13:59.821859 21204 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:59.828306 24698 consensus_queue.cc:1035] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:13:59.805853 24807 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:59.830178 24788 raft_consensus.cc:2781] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260812 19:13:59.830667 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 148847us
I20260812 19:13:59.832554 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:59.835255 24711 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:59.837702 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:13:59.838337 24826 log.cc:924] Time spent T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 1.572s user 0.010s sys 0.000s
I20260812 19:13:59.848043 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:13:59.863590 22052 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:13:59.825752 24674 raft_consensus.cc:2781] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:13:59.855026 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.855026 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:59.857048 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.819869 24872 raft_consensus.cc:2726] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:13:59.850960 21204 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:13:59.826076 24785 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:13:59.862906 22048 raft_consensus.cc:2454] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 3.
I20260812 19:13:59.873112 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:59.875528 21204 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:13:59.875398 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:13:59.885751 21329 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:59.886472 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.820097 24830 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:13:59.818670 24895 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:13:59.879238 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:59.915057 24895 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:13:59.915414 21152 raft_consensus.cc:2354] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4.
I20260812 19:13:59.916065 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:13:59.916504 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:59.861691 24807 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:13:59.893085 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:59.855105 24844 raft_consensus.cc:2781] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:13:59.887609 21318 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:13:59.936493 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.854820 24884 raft_consensus.cc:2726] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260812 19:13:59.907527 24834 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:13:59.908888 24837 consensus_meta.cc:287] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.744s user 0.010s sys 0.004s
W20260812 19:13:59.912330 22008 leader_election.cc:334] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.865s (SENT)
I20260812 19:13:59.950879 22008 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:13:59.951524 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:13:59.922463 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:59.951967 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:59.952750 21152 raft_consensus.cc:2379] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:13:59.861460 24788 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:59.931536 21329 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:13:59.905314 24744 raft_consensus.cc:2781] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:13:59.876250 24674 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:59.848224 24574 consensus_queue.cc:1035] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20260812 19:13:59.840446 24793 consensus_queue.cc:1035] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260812 19:13:59.891531 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:59.965477 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:13:59.953760 21153 raft_consensus.cc:1261] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260812 19:13:59.956705 21327 raft_consensus.cc:2396] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260812 19:13:59.868979 24794 consensus_queue.cc:1035] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
I20260812 19:13:59.916250 24795 raft_consensus.cc:2726] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:14:00.017148 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:00.032289 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:13:59.882800 24711 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.055320 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:00.037393 21244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57682 (request call id 337) took 1018 ms. Trace:
I20260812 19:13:59.964807 24744 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:13:59.956279 24787 raft_consensus.cc:2726] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:13:59.950523 24844 raft_consensus.cc:686] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:13:59.960350 24788 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:00.047374 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:00.015743 24674 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:00.060524 22008 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:00.056922 21244 rpcz_store.cc:276] 0812 19:13:59.018947 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:59.019284 (+ 337us) service_pool.cc:227] Handling call
0812 19:14:00.037145 (+1017861us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b5400150254. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:00.037317 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36881152}
I20260812 19:14:00.020744 24418 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:00.085305 21156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 2029) took 1250 ms. Trace:
I20260812 19:14:00.089540 21151 raft_consensus.cc:2379] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:00.093048 22008 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:14:00.064759 21287 connection.cc:232] RPC call timeout handler was delayed by -3.3574612149968743s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.205s for a response
I20260812 19:14:00.071177 24744 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:00.098758 21156 rpcz_store.cc:276] 0812 19:13:58.840235 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:58.847240 (+ 7005us) service_pool.cc:227] Handling call
0812 19:13:59.538264 (+691024us) spinlock_profiling.cc:243] Waited 3.3 ms on lock 0x7b5800001898. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52bce 000000000034aea9 00007f52de972a30 00007f52dea387b4 00007f52dea38821 00007f52de9fc2b0 00007f52de9e77f9 00007f52de9e72a2 00007f52d80d9684
0812 19:13:59.538459 (+ 195us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:59.644697 (+106238us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:59.644745 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:59.644787 (+ 42us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:13:59.644827 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:00.084936 (+440109us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b54002cc754. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:00.085022 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:00.085217 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":251469056}
I20260812 19:14:00.079836 22050 raft_consensus.cc:1261] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260812 19:14:00.116452 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:00.117233 21244 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:00.108495 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:00.118675 21151 raft_consensus.cc:2354] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 8. Current term is 9.
I20260812 19:14:00.120594 21151 raft_consensus.cc:1261] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260812 19:14:00.036432 24672 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.128355 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:14:00.115624 24606 consensus_meta.cc:287] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.685s user 0.007s sys 0.009s
W20260812 19:14:00.100683 24492 consensus_meta.cc:287] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.828s user 0.013s sys 0.001s
I20260812 19:14:00.132682 21241 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
W20260812 19:14:00.108161 24088 consensus_meta.cc:287] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.851s user 0.019s sys 0.001s
I20260812 19:14:00.143857 21149 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:00.093786 24836 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:00.108059 24787 raft_consensus.cc:2726] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260812 19:14:00.131767 24897 log.cc:924] Time spent T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
W20260812 19:14:00.160492 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 67522.5us
I20260812 19:14:00.162948 22008 leader_election.cc:398] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285) with higher term. Message: Invalid argument: T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:14:00.164189 22008 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:00.135012 24812 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:00.136592 24506 raft_consensus.cc:2726] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 19:14:00.172729 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 310784us
W20260812 19:14:00.143966 24783 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.639s user 0.024s sys 0.007s
W20260812 19:14:00.180163 24783 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.675s user 0.024s sys 0.008s
I20260812 19:14:00.190357 21149 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:14:00.200709 21203 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:00.179572 24836 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:00.218510 24836 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:00.219718 21203 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:00.223560 24678 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:00.224313 24678 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:00.227373 24638 consensus_meta.cc:287] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.587s user 0.014s sys 0.000s
I20260812 19:14:00.227896 24638 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:00.228439 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:00.229286 21152 raft_consensus.cc:2354] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7.
I20260812 19:14:00.230191 24638 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:00.313925 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:14:00.208611 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 260819us
I20260812 19:14:00.347564 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:00.348709 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:00.347260 21244 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:14:00.349699 21327 raft_consensus.cc:2396] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260812 19:14:00.189509 24787 raft_consensus.cc:2726] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:14:00.143414 24492 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.281226 24568 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:14:00.301800 24843 consensus_meta.cc:287] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.254s user 0.019s sys 0.001s
I20260812 19:14:00.155226 24088 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:00.315608 24453 consensus_meta.cc:287] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.652s user 0.014s sys 0.001s
I20260812 19:14:00.249018 21203 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:14:00.298386 24681 consensus_meta.cc:287] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.672s user 0.014s sys 0.002s
I20260812 19:14:00.249840 24836 raft_consensus.cc:2781] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:14:00.312916 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:00.382700 24453 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:00.335922 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 203121us
I20260812 19:14:00.384068 22050 raft_consensus.cc:2354] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6.
W20260812 19:14:00.384580 21287 connection.cc:232] RPC call timeout handler was delayed by -3.2755612150067464s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2141s for a response
W20260812 19:14:00.384991 21287 connection.cc:232] RPC call timeout handler was delayed by -3.2719612149521708s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21450000000000002s for a response
W20260812 19:14:00.385783 21287 connection.cc:232] RPC call timeout handler was delayed by -3.2197612149175256s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2203s for a response
W20260812 19:14:00.385908 24850 log.cc:924] Time spent T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 1.881s user 0.003s sys 0.007s
I20260812 19:14:00.386080 24453 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:00.386264 21287 connection.cc:232] RPC call timeout handler was delayed by -2.5152624429902062s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1603s for a response
W20260812 19:14:00.386801 21287 connection.cc:232] RPC call timeout handler was delayed by -2.4099624430527911s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17200000000000001s for a response
I20260812 19:14:00.387104 21150 raft_consensus.cc:2354] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 6. Current term is 7.
I20260812 19:14:00.388101 24576 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:00.388819 24576 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.390799 24626 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:14:00.390818 21287 connection.cc:232] RPC call timeout handler was delayed by -2.1030624429695308s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2061s for a response
I20260812 19:14:00.391530 24626 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:14:00.391610 21287 connection.cc:232] RPC call timeout handler was delayed by -1.9392624429892749s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2243s for a response
W20260812 19:14:00.393513 21287 connection.cc:232] RPC call timeout handler was delayed by -1.83135227591265s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18380000000000002s for a response
I20260812 19:14:00.393916 24626 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:00.395534 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:00.396309 21328 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 7
W20260812 19:14:00.397847 21154 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 2027) took 1571 ms (client timeout 2075 ms). Trace:
W20260812 19:14:00.398634 24531 consensus_meta.cc:287] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.753s user 0.016s sys 0.000s
I20260812 19:14:00.399313 24531 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.402071 24531 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:00.402808 22056 raft_consensus.cc:2354] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7.
I20260812 19:14:00.403123 21236 raft_consensus.cc:2354] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5.
I20260812 19:14:00.368007 24843 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:00.397879 24744 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.406558 21240 raft_consensus.cc:1226] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 7. Current term is 8. Ops: []
I20260812 19:14:00.374596 24568 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:00.398329 21154 rpcz_store.cc:269] 0812 19:13:58.826914 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:58.827076 (+ 162us) service_pool.cc:227] Handling call
0812 19:14:00.397664 (+1570588us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b5400214154. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:00.397782 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":313293568}
W20260812 19:14:00.394163 21287 connection.cc:232] RPC call timeout handler was delayed by -1.6873522759415209s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1998s for a response
W20260812 19:14:00.411396 21287 connection.cc:232] RPC call timeout handler was delayed by -1.4614522758638486s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22490000000000002s for a response
W20260812 19:14:00.233388 24853 consensus_meta.cc:287] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 1.101s user 0.011s sys 0.006s
I20260812 19:14:00.201159 24793 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:00.247427 21237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1720) took 1004 ms. Trace:
I20260812 19:14:00.248615 24720 raft_consensus.cc:2726] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:14:00.299008 24812 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:14:00.198052 24819 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:14:00.235390 24901 log.cc:924] Time spent T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260812 19:14:00.167557 22008 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:00.249094 24711 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:00.236363 24672 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:00.441694 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:00.443485 22045 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:14:00.230437 24678 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:00.231761 24662 consensus_meta.cc:287] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.595s user 0.010s sys 0.005s
I20260812 19:14:00.351994 21324 raft_consensus.cc:2396] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260812 19:14:00.446448 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:00.421128 24843 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:00.300696 24783 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:00.374007 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:00.356308 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:00.379806 24606 raft_consensus.cc:2726] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:00.380141 24830 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:00.374797 24681 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:00.382952 24895 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.405701 21235 raft_consensus.cc:2354] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6.
I20260812 19:14:00.405614 24492 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:14:00.290693 24877 consensus_meta.cc:287] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.659s user 0.011s sys 0.007s
I20260812 19:14:00.449424 24711 raft_consensus.cc:2726] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:14:00.183687 24844 consensus_queue.cc:237] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.478219 22056 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40598 (request call id 713) took 1317 ms. Trace:
I20260812 19:14:00.474176 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:00.479095 21319 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 7
W20260812 19:14:00.479692 21155 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 2020) took 2377 ms (client timeout 1686 ms). Trace:
I20260812 19:14:00.423606 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:00.424194 24576 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:00.423695 24088 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:00.480167 21155 rpcz_store.cc:269] 0812 19:13:58.102727 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:58.113647 (+ 10920us) service_pool.cc:227] Handling call
0812 19:14:00.479464 (+2365817us) spinlock_profiling.cc:243] Waited 828 ms on lock 0x7b54001239d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:00.479617 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1904971008}
I20260812 19:14:00.480786 22048 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 7
I20260812 19:14:00.425985 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:00.426152 24788 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:00.428447 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:00.429415 24744 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:00.428323 24807 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:00.429873 21287 connection.cc:232] RPC call timeout handler was delayed by -1.194962442968972s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.307s for a response
I20260812 19:14:00.430953 21115 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
W20260812 19:14:00.495645 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 64805.5us
I20260812 19:14:00.442018 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:00.499271 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:00.448870 21328 raft_consensus.cc:2454] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
W20260812 19:14:00.449054 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 281625us
I20260812 19:14:00.449605 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:00.436913 21237 rpcz_store.cc:276] 0812 19:13:59.254357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:59.254504 (+ 147us) service_pool.cc:227] Handling call
0812 19:13:59.271801 (+ 17297us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:59.291036 (+ 19235us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:59.291089 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:59.647732 (+356643us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b2c00000238. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d8495e11 00007f52d84a5442 00007f52d9ae1c11 00007f52d9aa658a 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772
0812 19:13:59.648194 (+ 462us) log.cc:842] Serialized 26 byte log entry
0812 19:14:00.102565 (+454371us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:14:00.130275 (+ 27710us) log.cc:842] Serialized 14 byte log entry
0812 19:14:00.130632 (+ 357us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:00.130749 (+ 117us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:00.246914 (+116165us) raft_consensus.cc:1671] finished
0812 19:14:00.247094 (+ 180us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:00.247323 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48533,"prepare.queue_time_us":20781,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":57,"raft.queue_time_us":19622,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":19167,"spinlock_wait_cycles":2785536,"thread_start_us":453964,"threads_started":1,"wal-append.queue_time_us":528953}
I20260812 19:14:00.350220 24834 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:00.452427 21327 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:00.459066 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:14:00.462713 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 213767us
I20260812 19:14:00.509948 22045 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
W20260812 19:14:00.317072 24832 consensus_meta.cc:287] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 1.183s user 0.008s sys 0.007s
I20260812 19:14:00.444610 24568 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:00.436965 24853 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:00.458334 24783 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:00.469113 24895 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:00.469298 24681 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:00.469944 21157 raft_consensus.cc:2354] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7.
I20260812 19:14:00.470424 22047 raft_consensus.cc:1226] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
W20260812 19:14:00.470647 24906 log.cc:924] Time spent T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20260812 19:14:00.441656 24819 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:00.476297 24830 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:00.441797 24812 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.445586 24662 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.420516 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:00.542987 21154 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:00.424479 21153 raft_consensus.cc:2379] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 3: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:00.478716 22056 rpcz_store.cc:276] 0812 19:13:59.161278 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:59.161506 (+ 228us) service_pool.cc:227] Handling call
0812 19:14:00.477950 (+1316444us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b540064c8d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:00.478146 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2732711296}
I20260812 19:14:00.484338 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:14:00.462484 24629 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.694s user 0.009s sys 0.018s
I20260812 19:14:00.446698 24793 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:00.507854 21112 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:00.547770 21204 leader_election.cc:398] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Vote denied by peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579) with higher term. Message: Invalid argument: T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7.
I20260812 19:14:00.512672 22008 leader_election.cc:398] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Vote denied by peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285) with higher term. Message: Invalid argument: T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:14:00.515077 22048 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:00.521098 21288 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:14:00.522323 21319 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:00.523409 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:00.532727 21115 leader_election.cc:398] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Vote denied by peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579) with higher term. Message: Invalid argument: T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6.
I20260812 19:14:00.550244 21320 raft_consensus.cc:2379] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:14:00.509397 24606 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:00.493839 22050 raft_consensus.cc:3037] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 5
I20260812 19:14:00.498528 24088 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260812 19:14:00.541494 22053 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 LEARNER]: Advancing to term 9
I20260812 19:14:00.558490 21287 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:00.509155 22046 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:00.548141 21204 leader_election.cc:401] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:14:00.492417 24836 raft_consensus.cc:686] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:00.511581 24711 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:00.550420 21115 leader_election.cc:401] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260812 19:14:00.551184 22008 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:14:00.552120 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 44443.6us
I20260812 19:14:00.546777 21244 raft_consensus.cc:2362] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6. Re-sending same reply.
I20260812 19:14:00.567108 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:14:00.550480 24629 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.782s user 0.009s sys 0.018s
I20260812 19:14:00.568930 21112 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:00.579433 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1725) took 1316 ms. Trace:
I20260812 19:14:00.579833 21239 rpcz_store.cc:276] 0812 19:13:59.262513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:59.275363 (+ 12850us) service_pool.cc:227] Handling call
0812 19:14:00.579154 (+1303791us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b5400304154. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:00.579360 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":248533376}
I20260812 19:14:00.580986 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:00.581470 21112 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:00.581704 21239 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:14:00.570013 22008 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:00.582830 24088 raft_consensus.cc:2781] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:00.583431 24088 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:00.582878 24711 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.583797 24088 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:00.585553 24711 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:00.585657 24672 raft_consensus.cc:2781] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:14:00.586349 24672 raft_consensus.cc:686] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:00.578346 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:00.587546 24672 consensus_queue.cc:237] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:00.588799 21233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1726) took 1326 ms. Trace:
I20260812 19:14:00.589237 21233 rpcz_store.cc:276] 0812 19:13:59.262823 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:59.410349 (+147526us) service_pool.cc:227] Handling call
0812 19:14:00.588631 (+1178282us) spinlock_profiling.cc:243] Waited 304 ms on lock 0x7b5400150ed4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:00.588739 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":699705600}
I20260812 19:14:00.568109 21115 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:14:00.592119 24819 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:00.593523 24553 raft_consensus.cc:2726] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 19:14:00.594101 24914 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:00.595219 24553 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:00.595759 24553 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.596107 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:00.597846 24553 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:00.599668 24793 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:00.600481 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:00.600744 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:00.601311 21151 raft_consensus.cc:2379] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 5: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:00.601421 21152 raft_consensus.cc:2454] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:14:00.602233 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:00.549897 24840 consensus_queue.cc:1035] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260812 19:14:00.526413 24832 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:00.586272 24711 raft_consensus.cc:2726] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 19:14:00.525938 24453 consensus_queue.cc:1035] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260812 19:14:00.524991 24834 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.576272 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1729) took 1311 ms. Trace:
I20260812 19:14:00.576762 24836 consensus_queue.cc:237] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:00.562561 24568 raft_consensus.cc:2781] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:14:00.568607 24292 raft_consensus.cc:2726] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:14:00.594727 24914 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.588856 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:00.570881 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:00.635291 21327 raft_consensus.cc:3037] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:14:00.574093 24813 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:00.536027 24674 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:00.560245 24681 raft_consensus.cc:2781] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election won for term 7
W20260812 19:14:00.604912 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 54338us
I20260812 19:14:00.605360 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:00.621763 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:00.621500 21238 rpcz_store.cc:276] 0812 19:13:59.264630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:59.500379 (+235749us) service_pool.cc:227] Handling call
0812 19:13:59.734132 (+233753us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:59.852402 (+118270us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:00.575877 (+723475us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b54003bf450. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:00.576159 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":535584000}
I20260812 19:14:00.624634 21322 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:00.645053 21237 raft_consensus.cc:1799] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Handling vote request from an unknown peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:14:00.645532 21237 raft_consensus.cc:2379] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 6: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:14:00.606009 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:00.615531 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:00.646111 21152 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 6
I20260812 19:14:00.649303 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:00.649981 21151 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:00.604663 24915 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
W20260812 19:14:00.629637 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 82011.6us
I20260812 19:14:00.632787 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:00.651842 21328 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:00.621248 24853 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:00.563359 24606 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.604470 24638 raft_consensus.cc:2726] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579) with higher term. Message: Invalid argument: T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6.
I20260812 19:14:00.638557 22053 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260812 19:14:00.644513 21239 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
I20260812 19:14:00.607779 21154 raft_consensus.cc:2454] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
I20260812 19:14:00.662285 21287 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:00.635838 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:00.629148 24568 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:00.660065 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:00.666491 21238 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:00.660928 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:00.667258 21235 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1731) took 1399 ms. Trace:
I20260812 19:14:00.664315 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:00.667680 21235 rpcz_store.cc:276] 0812 19:13:59.268274 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:59.360685 (+ 92411us) service_pool.cc:227] Handling call
0812 19:14:00.664294 (+1303609us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400255dd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:00.667179 (+ 2885us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2346013952}
I20260812 19:14:00.662809 21327 raft_consensus.cc:2454] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:14:00.663249 22011 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:14:00.668911 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:00.669720 21235 raft_consensus.cc:2379] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 7: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:00.670081 22011 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:00.633940 24843 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 7
I20260812 19:14:00.672408 21200 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:00.672377 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:00.673458 21235 raft_consensus.cc:2354] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:14:00.674041 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:00.674927 21327 raft_consensus.cc:2396] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260812 19:14:00.675153 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:00.676450 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:00.679566 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:00.648165 24895 raft_consensus.cc:2726] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:14:00.685197 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:00.686100 21237 raft_consensus.cc:2379] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 6: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:14:00.680753 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:00.682713 21154 raft_consensus.cc:1261] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260812 19:14:00.682910 22011 leader_election.cc:398] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:14:00.687181 22011 leader_election.cc:401] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:14:00.671103 24568 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:00.655794 24681 raft_consensus.cc:686] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:00.692230 22011 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:00.699930 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:00.701750 21327 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:00.703925 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:00.705991 21238 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:14:00.707161 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:00.707852 21238 raft_consensus.cc:2362] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8. Re-sending same reply.
I20260812 19:14:00.692404 21151 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 9.
I20260812 19:14:00.689178 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:00.709205 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:00.709867 21151 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:00.710065 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:00.712165 21154 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260812 19:14:00.711021 22011 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:00.712472 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:00.713315 21237 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:00.679804 24853 raft_consensus.cc:2726] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:00.684523 24492 raft_consensus.cc:2781] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:14:00.684278 24711 raft_consensus.cc:2781] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:00.735562 24711 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:00.735054 24681 consensus_queue.cc:237] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:00.735921 24606 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:00.736033 24711 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:00.736804 24662 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 9 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:00.737433 22050 raft_consensus.cc:2454] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:14:00.738435 24492 raft_consensus.cc:686] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:14:00.739132 24088 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:00.739497 24492 consensus_queue.cc:237] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.741442 24088 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:00.683854 24838 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:00.685055 24916 raft_consensus.cc:2781] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:14:00.716135 21328 raft_consensus.cc:2454] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:14:00.696062 24788 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:00.744179 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:00.737496 24662 raft_consensus.cc:2781] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:14:00.757707 21151 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
W20260812 19:14:00.763212 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 47775.4us
I20260812 19:14:00.764381 21327 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:14:00.703816 24787 raft_consensus.cc:2726] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:00.684688 24629 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20260812 19:14:00.760756 21237 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 9.
I20260812 19:14:00.663199 24915 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:00.794579 24917 log.cc:924] Time spent T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
I20260812 19:14:00.796542 24744 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:00.797396 24744 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:00.800247 24922 log.cc:924] Time spent T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.139s user 0.001s sys 0.000s
I20260812 19:14:00.745101 24913 raft_consensus.cc:2726] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 4. Current term is 5.
I20260812 19:14:00.772475 24812 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:00.785784 24787 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:00.804059 24787 raft_consensus.cc:504] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.815081 24787 leader_election.cc:288] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:00.762943 24830 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
W20260812 19:14:00.899453 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 188503us
I20260812 19:14:00.902791 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:00.904048 21238 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:00.911713 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:00.929782 21152 raft_consensus.cc:2454] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:14:00.783437 24916 raft_consensus.cc:686] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:00.943243 24711 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:00.893997 24783 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:00.762853 24838 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:00.966089 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:14:01.181351 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 279081us
I20260812 19:14:01.201220 21203 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:01.220810 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:01.182504 21154 raft_consensus.cc:1226] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 7. Current term is 8. Ops: []
I20260812 19:14:01.174300 21287 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:01.222884 22011 leader_election.cc:398] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 6. Current term is 7.
I20260812 19:14:01.213842 21322 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:01.223855 22011 leader_election.cc:401] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20260812 19:14:01.224007 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 49827.9us
I20260812 19:14:01.213164 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:01.209371 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:01.225848 21150 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
W20260812 19:14:01.181833 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 305151us
W20260812 19:14:01.226940 22051 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.001s sys 0.014s
W20260812 19:14:01.227402 22051 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.533s user 0.002s sys 0.014s
I20260812 19:14:01.227638 21203 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:14:00.965757 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:01.178800 24825 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:01.249526 24825 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:01.252349 24825 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:01.253240 24844 consensus_queue.cc:1046] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260812 19:14:01.249774 22052 raft_consensus.cc:1226] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260812 19:14:00.988641 24807 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:01.258493 24812 raft_consensus.cc:2781] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:14:01.259023 24812 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:01.259359 24812 raft_consensus.cc:3037] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:14:01.260036 21238 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:14:01.261799 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 350) took 1371 ms. Trace:
I20260812 19:14:01.267971 24838 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:00.945382 24832 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:01.262166 21238 rpcz_store.cc:276] 0812 19:13:59.889932 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:01.260867 (+1370935us) service_pool.cc:227] Handling call
0812 19:14:01.261223 (+ 356us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:01.261421 (+ 198us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:01.261448 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:01.261493 (+ 45us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:14:01.261523 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:01.261598 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:01.261707 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:00.937608 24830 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:01.271799 24832 raft_consensus.cc:3032] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 7 LEADER]: Stepping down as leader of term 7
I20260812 19:14:01.274936 24832 raft_consensus.cc:729] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:01.276293 24832 consensus_queue.cc:260] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:01.279837 22046 consensus_meta.cc:287] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.675s user 0.012s sys 0.000s
I20260812 19:14:01.280319 22046 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 8.
I20260812 19:14:01.278355 21150 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:14:01.281085 22053 raft_consensus.cc:1226] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260812 19:14:01.284287 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:01.286022 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:01.284472 24830 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:01.288859 21153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1692) took 1241 ms. Trace:
I20260812 19:14:01.293543 21153 rpcz_store.cc:276] 0812 19:14:00.047345 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:00.047989 (+ 644us) service_pool.cc:227] Handling call
0812 19:14:01.288595 (+1240606us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b54000c4754. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:01.288777 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":846477056}
I20260812 19:14:01.287045 21238 raft_consensus.cc:1226] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 8. Current term is 9. Ops: [4.6-8.8]
I20260812 19:14:01.177517 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:01.035871 24674 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:01.294793 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:01.296957 21328 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:01.026371 24914 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:01.215518 24834 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:01.210289 24606 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:14:01.278653 24568 consensus_meta.cc:287] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.555s user 0.011s sys 0.001s
I20260812 19:14:01.228891 24574 raft_consensus.cc:2726] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:00.963321 24662 raft_consensus.cc:686] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:14:01.228894 24853 raft_consensus.cc:2726] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260812 19:14:01.296620 24813 consensus_meta.cc:287] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.656s user 0.009s sys 0.005s
I20260812 19:14:01.308519 21238 raft_consensus.cc:1226] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260812 19:14:01.030658 24916 consensus_queue.cc:237] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:01.308745 24606 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:01.309335 24932 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:01.304011 22050 raft_consensus.cc:2454] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:14:01.309041 24568 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:01.308881 24914 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:01.331655 24568 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:01.332422 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 2067) took 1003 ms. Trace:
I20260812 19:14:01.332902 21318 rpcz_store.cc:276] 0812 19:14:00.328560 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:00.329310 (+ 750us) service_pool.cc:227] Handling call
0812 19:14:00.329703 (+ 393us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:00.407743 (+ 78040us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:00.407774 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:00.407820 (+ 46us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:14:00.407860 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:01.332134 (+924274us) spinlock_profiling.cc:243] Waited 77.8 ms on lock 0x7b54002314d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:01.332185 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:01.332356 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":178854144}
I20260812 19:14:01.323572 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37894 (request call id 1619) took 1006 ms. Trace:
I20260812 19:14:01.328716 21319 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260812 19:14:01.331501 24914 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:01.349960 21238 rpcz_store.cc:276] 0812 19:14:00.317623 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:01.296943 (+979320us) service_pool.cc:227] Handling call
0812 19:14:01.308253 (+ 11310us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:01.323301 (+ 15048us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:01.323519 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:01.353168 24915 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:01.363730 24711 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:01.365408 24662 consensus_queue.cc:237] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:01.367555 24606 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:01.368281 24744 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:01.368461 24914 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:01.316264 24931 raft_consensus.cc:2726] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 6. Current term is 7.
I20260812 19:14:01.371544 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 349) took 1482 ms. Trace:
W20260812 19:14:01.371750 21322 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1821) took 2172 ms (client timeout 2091 ms). Trace:
W20260812 19:14:01.372201 21322 rpcz_store.cc:269] 0812 19:13:59.199589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:59.199797 (+ 208us) service_pool.cc:227] Handling call
0812 19:14:01.371520 (+2171723us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b5400083254. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:01.371645 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3174914432}
I20260812 19:14:01.371995 21239 rpcz_store.cc:276] 0812 19:13:59.889626 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:00.720586 (+830960us) service_pool.cc:227] Handling call
0812 19:14:00.729124 (+ 8538us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:00.729553 (+ 429us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:00.729583 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:00.730055 (+ 472us) log.cc:842] Serialized 26 byte log entry
0812 19:14:00.899185 (+169130us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:14:00.943216 (+ 44031us) log.cc:842] Serialized 14 byte log entry
0812 19:14:00.944458 (+ 1242us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:00.944544 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:01.371313 (+426769us) raft_consensus.cc:1671] finished
0812 19:14:01.371337 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:01.371477 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":912,"prepare.queue_time_us":44116,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":20052,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":201,"thread_start_us":168682,"threads_started":1,"wal-append.queue_time_us":572637}
I20260812 19:14:01.307873 24674 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:01.415097 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:01.416349 21322 raft_consensus.cc:2362] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7. Re-sending same reply.
I20260812 19:14:01.419605 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:01.420617 21322 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 8
W20260812 19:14:01.421221 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 195985us
I20260812 19:14:01.422171 21112 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:01.424427 21319 raft_consensus.cc:1261] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260812 19:14:01.427021 21238 raft_consensus.cc:1226] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260812 19:14:01.428604 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37894 (request call id 1621) took 1110 ms. Trace:
I20260812 19:14:01.428944 21238 rpcz_store.cc:276] 0812 19:14:00.318266 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:01.415458 (+1097192us) service_pool.cc:227] Handling call
0812 19:14:01.426724 (+ 11266us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:01.428316 (+ 1592us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:01.428547 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:14:01.438910 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 211340us
I20260812 19:14:01.440972 21203 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:01.430125 21328 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:14:01.446283 21318 raft_consensus.cc:1226] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 7. Current term is 8. Ops: []
I20260812 19:14:01.446682 21112 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:01.469867 21203 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:14:01.427963 24674 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:01.476330 24794 consensus_queue.cc:1035] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:14:01.479236 24794 raft_consensus.cc:2781] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader election won for term 9
W20260812 19:14:01.480230 24843 consensus_meta.cc:287] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.795s user 0.013s sys 0.000s
I20260812 19:14:01.480754 24843 raft_consensus.cc:2726] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579) with higher term. Message: Invalid argument: T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7.
I20260812 19:14:01.353016 24932 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:01.483603 24932 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:01.484375 21287 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:01.485953 21287 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:01.486688 24681 raft_consensus.cc:2781] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:14:01.480896 24794 raft_consensus.cc:686] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 9 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:14:01.490869 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 44350.7us
I20260812 19:14:01.496537 21115 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:14:01.498216 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1732) took 2229 ms. Trace:
I20260812 19:14:01.503192 21243 rpcz_store.cc:276] 0812 19:13:59.268652 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:59.638889 (+370237us) service_pool.cc:227] Handling call
0812 19:13:59.639279 (+ 390us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:59.639717 (+ 438us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:59.639745 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:59.640270 (+ 525us) log.cc:842] Serialized 26 byte log entry
0812 19:13:59.917742 (+277472us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:13:59.960302 (+ 42560us) log.cc:842] Serialized 14 byte log entry
0812 19:13:59.960690 (+ 388us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:59.960795 (+ 105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:01.495346 (+1534551us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b18002c0200. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a638 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
0812 19:14:01.497873 (+ 2527us) raft_consensus.cc:1671] finished
0812 19:14:01.498003 (+ 130us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:01.498147 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2306,"prepare.queue_time_us":40664,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":3297,"raft.queue_time_us":52179,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":13278,"spinlock_wait_cycles":24029440,"thread_start_us":277053,"threads_started":1,"wal-append.queue_time_us":423519}
W20260812 19:14:01.613878 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 109023us
W20260812 19:14:01.614389 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 117954us
I20260812 19:14:01.615744 21203 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:01.508591 21235 raft_consensus.cc:2454] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
W20260812 19:14:01.509815 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 287605us
I20260812 19:14:01.626758 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:01.630792 24492 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:01.630962 24606 raft_consensus.cc:2781] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:14:01.631878 24915 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:01.632423 24843 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260812 19:14:01.632627 24915 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:01.633949 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:01.634148 24932 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:14:01.494561 24944 log.cc:924] Time spent T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.072s user 0.002s sys 0.000s
I20260812 19:14:01.637207 24453 raft_consensus.cc:2781] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:14:01.637485 24943 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:01.638013 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:01.497468 24938 consensus_queue.cc:1035] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.196s
I20260812 19:14:01.638384 24943 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:01.640384 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:01.641304 24943 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:01.497427 24820 consensus_queue.cc:1046] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.145s
I20260812 19:14:01.511643 24681 raft_consensus.cc:686] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:01.637739 24453 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:01.633119 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:01.636276 24683 consensus_queue.cc:1046] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260812 19:14:01.648948 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:01.654829 21156 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
W20260812 19:14:01.680534 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 49603.9us
I20260812 19:14:01.688109 21203 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:14:01.641371 24889 log.cc:924] Time spent T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 2.024s user 0.008s sys 0.000s
I20260812 19:14:01.502197 24838 raft_consensus.cc:2741] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260812 19:14:01.506140 24674 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:01.642673 24820 raft_consensus.cc:3032] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Stepping down as leader of term 7
I20260812 19:14:01.642208 24943 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:01.700491 24820 raft_consensus.cc:729] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:01.700953 24943 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:14:01.701634 24820 consensus_queue.cc:260] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:01.500519 24916 raft_consensus.cc:2726] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260812 19:14:01.699630 24629 consensus_queue.cc:1046] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:14:01.698588 24606 raft_consensus.cc:686] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:14:01.702826 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 278575us
I20260812 19:14:01.495100 24841 raft_consensus.cc:2781] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader election won for term 8
I20260812 19:14:01.643874 24843 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:01.680369 24453 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:01.656826 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:01.728719 21156 raft_consensus.cc:1261] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260812 19:14:01.711199 24795 raft_consensus.cc:2781] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:14:01.757342 24915 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:01.757138 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 271255us
I20260812 19:14:01.758386 22011 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:01.756016 21322 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:14:01.782838 24662 raft_consensus.cc:2781] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:14:01.783327 24662 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:01.783557 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1851) took 1178 ms. Trace:
I20260812 19:14:01.783713 24662 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:01.784053 21328 rpcz_store.cc:276] 0812 19:14:00.604839 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:01.761848 (+1157009us) service_pool.cc:227] Handling call
0812 19:14:01.782998 (+ 21150us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:01.783207 (+ 209us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:01.783239 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:01.783275 (+ 36us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:14:01.783306 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:01.783363 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:01.783499 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:01.784749 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 2075) took 1233 ms. Trace:
I20260812 19:14:01.785131 21318 rpcz_store.cc:276] 0812 19:14:00.551682 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:01.358002 (+806320us) service_pool.cc:227] Handling call
0812 19:14:01.445988 (+ 87986us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:01.462701 (+ 16713us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:01.784521 (+321820us) spinlock_profiling.cc:243] Waited 87.5 ms on lock 0x7b5400230ad0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:01.784685 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":201257856}
I20260812 19:14:01.786935 24942 consensus_queue.cc:1035] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260812 19:14:01.788712 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1734) took 2519 ms. Trace:
I20260812 19:14:01.789147 21240 rpcz_store.cc:276] 0812 19:13:59.269273 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:59.665411 (+396138us) service_pool.cc:227] Handling call
0812 19:14:00.038541 (+373130us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:00.407002 (+368461us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:01.787943 (+1380941us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b5400150254. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:01.788488 (+ 545us) spinlock_profiling.cc:243] Waited 373 ms on lock 0x7b5400150250. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:01.788642 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1703057408}
I20260812 19:14:01.789978 24853 raft_consensus.cc:2741] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260812 19:13:59.218122 21321 rpcz_store.cc:276] 0812 19:13:58.140203 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:58.140427 (+ 224us) service_pool.cc:227] Handling call
0812 19:13:58.187161 (+ 46734us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:13:58.187418 (+ 257us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:13:58.933620 (+746202us) log.cc:842] Serialized 14 byte log entry
0812 19:13:59.216316 (+282696us) log.cc:842] Serialized 14 byte log entry
0812 19:13:59.216795 (+ 479us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:13:59.216913 (+ 118us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:13:59.217003 (+ 90us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:13:59.217149 (+ 146us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:13:59.217393 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":851,"raft.queue_time_us":8640,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":7410,"thread_start_us":267573,"threads_started":1}
W20260812 19:14:01.773422 24941 log.cc:924] Time spent T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.138s user 0.000s sys 0.000s
W20260812 19:14:01.777194 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 2.37188e+06us
I20260812 19:14:01.798216 24943 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:01.798841 21114 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:14:01.799901 24606 consensus_queue.cc:237] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:01.801769 21114 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:01.716492 24674 raft_consensus.cc:3032] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Stepping down as leader of term 7
I20260812 19:14:01.807976 21112 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:01.631487 24492 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:01.810758 21287 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:14:01.833316 21287 leader_election.cc:398] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 7. Current term is 8.
I20260812 19:14:01.834865 24844 raft_consensus.cc:2741] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 3 LEADER]: Leader election decision vote started in defunct term 2: won
I20260812 19:14:01.830250 24883 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:14:01.836900 24883 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:01.841042 24681 consensus_queue.cc:237] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:01.845705 24683 raft_consensus.cc:2781] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:14:01.725462 24841 raft_consensus.cc:686] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 8 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:01.731896 24820 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:01.682499 24932 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:01.800520 21157 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1683) took 1910 ms (client timeout 1952 ms). Trace:
W20260812 19:14:01.849244 21157 rpcz_store.cc:269] 0812 19:13:59.890452 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:59.892684 (+ 2232us) service_pool.cc:227] Handling call
0812 19:14:01.800290 (+1907606us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x7b5400214654. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:01.800453 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1250142464}
I20260812 19:14:01.836946 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:01.853727 21318 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 9.
I20260812 19:14:01.492624 24794 consensus_queue.cc:237] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:01.852630 21287 leader_election.cc:398] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4.
I20260812 19:14:01.855943 21287 leader_election.cc:401] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:14:01.857746 21287 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:01.862039 24838 raft_consensus.cc:2726] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:14:01.831560 24674 raft_consensus.cc:729] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:01.831382 24418 raft_consensus.cc:2726] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:14:01.875928 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:01.879171 21157 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:14:01.863495 24674 consensus_queue.cc:260] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:01.847289 24795 raft_consensus.cc:686] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:01.882073 22011 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:01.751840 24843 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:01.873179 24943 raft_consensus.cc:2781] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader election won for term 8
I20260812 19:14:01.891358 21287 leader_election.cc:398] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 8. Current term is 9.
I20260812 19:14:01.899326 21287 leader_election.cc:401] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:14:01.901708 21287 leader_election.cc:398] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7.
I20260812 19:14:01.902139 21287 leader_election.cc:401] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:14:01.903065 24938 raft_consensus.cc:2726] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 8. Current term is 9.
I20260812 19:14:01.903656 24935 raft_consensus.cc:2781] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:14:01.904340 24935 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:14:01.903581 24841 consensus_queue.cc:237] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:01.903157 24795 consensus_queue.cc:237] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:01.904744 24935 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:01.912369 24662 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:01.878103 24838 consensus_queue.cc:1035] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260812 19:14:01.915735 24943 raft_consensus.cc:686] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 8 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:01.916003 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:01.916728 21157 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 7
I20260812 19:14:01.917322 24943 consensus_queue.cc:237] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:01.877887 24844 raft_consensus.cc:2726] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4.
I20260812 19:14:01.932267 22008 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:01.932144 21157 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
I20260812 19:14:01.876200 22050 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:14:01.943699 22011 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:14:01.948900 21158 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54790 (request call id 2049) took 1351 ms (client timeout 1517 ms). Trace:
W20260812 19:14:01.949541 21158 rpcz_store.cc:269] 0812 19:14:00.597558 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:00.601978 (+ 4420us) service_pool.cc:227] Handling call
0812 19:14:01.948677 (+1346699us) spinlock_profiling.cc:243] Waited 840 ms on lock 0x7b5400325854. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:01.948833 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1932120832}
I20260812 19:14:01.951790 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:01.952623 21239 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:14:01.952927 21243 raft_consensus.cc:1261] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260812 19:14:01.952010 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:01.958233 21238 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
W20260812 19:14:01.962852 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 152911us
W20260812 19:14:01.946311 24812 consensus_meta.cc:287] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.686s user 0.013s sys 0.000s
I20260812 19:14:01.941504 24838 raft_consensus.cc:2726] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7.
I20260812 19:14:01.966995 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:01.946585 24674 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:01.944986 24940 raft_consensus.cc:2781] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:14:01.946820 24942 raft_consensus.cc:2726] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:01.969738 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:01.949107 24883 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:01.903863 24938 consensus_queue.cc:1035] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20260812 19:14:02.001132 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:14:02.002408 21244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1603) took 2122 ms (client timeout 2043 ms). Trace:
W20260812 19:14:02.002948 21244 rpcz_store.cc:269] 0812 19:13:59.880451 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:00.587656 (+707205us) service_pool.cc:227] Handling call
0812 19:14:02.000931 (+1413275us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b5800004598. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:14:02.002309 (+ 1378us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40273280}
I20260812 19:14:01.984102 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:01.981693 24662 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:01.970099 24812 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:02.006320 24942 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
W20260812 19:14:02.035696 21203 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.70.129:36551 timed out after 1.517s (SENT) [suppressed 32 similar messages]
I20260812 19:14:02.019686 24940 raft_consensus.cc:686] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [term 7 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:14:02.043278 24940 consensus_queue.cc:237] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:02.019361 24914 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:02.008607 24952 consensus_queue.cc:1035] T 7dfc261e3ffb48ddaa432f57bf7aad22 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260812 19:14:02.061789 24812 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:02.051625 24942 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:02.063422 21239 raft_consensus.cc:2379] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:14:02.064942 24492 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:02.066054 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 2079) took 1496 ms. Trace:
I20260812 19:14:02.065968 21244 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:14:02.066912 21244 raft_consensus.cc:595] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:14:02.067512 21244 raft_consensus.cc:603] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:14:02.066601 21319 rpcz_store.cc:276] 0812 19:14:00.569382 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:01.436558 (+867176us) service_pool.cc:227] Handling call
0812 19:14:01.437073 (+ 515us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:01.437560 (+ 487us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:01.437603 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:01.463552 (+ 25949us) log.cc:842] Serialized 26 byte log entry
0812 19:14:01.877221 (+413669us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:14:01.877261 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:01.877346 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:02.065771 (+188425us) raft_consensus.cc:1671] finished
0812 19:14:02.065800 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:02.065969 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2368,"raft.queue_time_us":128,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":20544,"thread_start_us":191409,"threads_started":1,"wal-append.queue_time_us":465977}
I20260812 19:14:02.065014 24942 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:02.079242 24932 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:02.059317 24914 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:02.229998 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 266196us
W20260812 19:14:02.236282 21320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 2046) took 3173 ms (client timeout 1573 ms). Trace:
W20260812 19:14:02.236932 21320 rpcz_store.cc:269] 0812 19:13:59.063099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:59.063270 (+ 171us) service_pool.cc:227] Handling call
0812 19:14:02.235987 (+3172717us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b54002319d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:02.236184 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3332213504}
I20260812 19:14:02.294495 21323 raft_consensus.cc:1226] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5. Ops: []
W20260812 19:14:02.298182 21200 leader_election.cc:334] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.573s (SENT)
I20260812 19:14:02.298873 21200 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:02.317381 24795 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
W20260812 19:14:02.350849 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 119674us
I20260812 19:14:02.352314 21114 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:14:02.233521 24932 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:02.295068 24794 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:02.105187 24683 raft_consensus.cc:686] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:02.525043 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:14:02.521517 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 223372us
I20260812 19:14:02.525424 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:14:02.536901 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 183240us
I20260812 19:14:02.566461 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:02.567080 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:02.525906 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:02.567780 21321 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:02.568634 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:02.568750 21114 leader_election.cc:398] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6.
I20260812 19:14:02.524672 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:02.569125 21114 leader_election.cc:401] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20260812 19:14:02.399166 21112 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.003s sys 0.011s
I20260812 19:14:02.527814 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:02.528133 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:14:02.569880 21112 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.760s user 0.004s sys 0.012s
W20260812 19:14:02.398689 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 363017us
I20260812 19:14:02.570844 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:02.572167 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:02.573324 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:02.573491 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:02.573745 21151 raft_consensus.cc:1261] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260812 19:14:02.574015 21154 raft_consensus.cc:2454] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 4.
I20260812 19:14:02.571846 21287 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:14:02.572269 21158 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:47804
W20260812 19:14:02.570312 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 761003us
I20260812 19:14:02.576005 21158 raft_consensus.cc:595] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:14:02.576380 21158 raft_consensus.cc:603] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:14:02.577795 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:02.580199 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:02.574224 21320 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:14:02.578711 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:02.581312 22011 leader_election.cc:334] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807): Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 1.853s (SENT)
I20260812 19:14:02.582106 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:02.582233 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:02.582849 21328 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:14:02.583204 22011 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:02.582368 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:02.522189 24954 log.cc:924] Time spent T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.447s user 0.000s sys 0.002s
W20260812 19:14:02.576704 24874 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.000s sys 0.014s
I20260812 19:14:02.666906 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:02.601168 24794 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:02.668475 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:02.669556 24794 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:02.669772 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:02.670504 21157 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:14:02.589982 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:02.666925 21322 raft_consensus.cc:2454] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:14:02.671617 21149 raft_consensus.cc:2379] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 6: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:02.583773 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:02.334444 24812 raft_consensus.cc:2726] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:14:02.617336 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:02.582983 21155 raft_consensus.cc:2454] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:14:02.581499 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:02.677251 24832 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:02.428346 24942 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:14:02.550848 24877 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:14:02.682391 24935 consensus_meta.cc:287] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.770s user 0.009s sys 0.003s
I20260812 19:14:02.361606 24681 raft_consensus.cc:2741] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260812 19:14:02.685277 24841 raft_consensus.cc:2726] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6.
I20260812 19:14:02.490906 24844 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:14:02.747044 24968 log.cc:924] Time spent T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.196s user 0.001s sys 0.000s
I20260812 19:14:02.734324 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:02.694531 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:02.747810 21155 raft_consensus.cc:3037] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:14:02.720939 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 149287us
W20260812 19:14:02.666827 24874 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s user 0.002s sys 0.024s
I20260812 19:14:02.738901 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:02.636399 24883 raft_consensus.cc:2726] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:14:02.493266 24843 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:02.744957 21152 raft_consensus.cc:2354] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 8. Current term is 9.
I20260812 19:14:02.746563 24844 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:14:02.666992 24966 log.cc:924] Time spent T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.304s user 0.001s sys 0.000s
I20260812 19:14:02.749182 21287 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:02.748775 24847 raft_consensus.cc:3032] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Stepping down as leader of term 7
I20260812 19:14:02.767375 21321 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
I20260812 19:14:02.746398 24984 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:02.581421 24932 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:02.326975 24795 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:02.777954 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:02.789937 21150 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 8.
I20260812 19:14:02.740037 24877 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:02.787815 21287 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:02.782325 24847 raft_consensus.cc:729] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:02.811018 21155 raft_consensus.cc:2454] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:14:02.579573 24965 consensus_queue.cc:1035] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.209s
I20260812 19:14:02.783203 24837 raft_consensus.cc:2741] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260812 19:14:02.893962 24976 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:02.894688 24976 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:02.895733 24820 consensus_meta.cc:287] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.040s user 0.013s sys 0.001s
I20260812 19:14:02.897285 24976 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:02.692895 24914 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:02.824168 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1849) took 2394 ms. Trace:
I20260812 19:14:02.826318 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:02.744710 24935 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:02.898833 24970 consensus_queue.cc:1035] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260812 19:14:02.899304 21243 raft_consensus.cc:2354] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7.
I20260812 19:14:02.827785 22047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 612) took 2404 ms. Trace:
I20260812 19:14:02.780500 24883 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:02.829907 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:02.832587 24938 raft_consensus.cc:2726] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:02.834668 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1820) took 3668 ms. Trace:
I20260812 19:14:02.836424 24787 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:14:02.837935 22050 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40624 (request call id 620) took 1210 ms. Trace:
I20260812 19:14:02.845762 24853 raft_consensus.cc:2726] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 19:14:02.855485 24936 log.cc:924] Time spent T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 1.220s user 0.007s sys 0.008s
W20260812 19:14:02.862438 21235 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57682 (request call id 353) took 2454 ms (client timeout 1809 ms). Trace:
W20260812 19:14:02.898660 24978 log.cc:924] Time spent T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
W20260812 19:14:02.868629 24927 log.cc:924] Time spent T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 1.366s user 0.007s sys 0.011s
I20260812 19:14:02.871837 21152 raft_consensus.cc:1226] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [term 8 LEADER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 7. Current term is 8. Ops: []
I20260812 19:14:02.878690 24874 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 19:14:02.879741 24843 consensus_queue.cc:1035] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260812 19:14:02.882030 24974 log.cc:924] Time spent T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.367s user 0.000s sys 0.001s
I20260812 19:14:02.882581 24795 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:02.882812 24959 log.cc:924] Time spent T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.310s user 0.001s sys 0.000s
W20260812 19:14:02.883663 24964 log.cc:924] Time spent T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.150s user 0.000s sys 0.000s
I20260812 19:14:02.743036 24942 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:02.888659 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 2066) took 2560 ms. Trace:
I20260812 19:14:02.792117 24984 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:02.897473 24453 consensus_meta.cc:287] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.141s user 0.015s sys 0.000s
I20260812 19:14:02.900518 22047 rpcz_store.cc:276] 0812 19:14:00.423080 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:00.424633 (+ 1553us) service_pool.cc:227] Handling call
0812 19:14:00.425034 (+ 401us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:00.530333 (+105299us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:02.827393 (+2297060us) spinlock_profiling.cc:243] Waited 45.2 ms on lock 0x7b5400620554. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:02.827656 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":103889664}
I20260812 19:14:02.920075 24935 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:02.898363 21324 rpcz_store.cc:276] 0812 19:14:00.429409 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:02.823097 (+2393688us) service_pool.cc:227] Handling call
0812 19:14:02.823532 (+ 435us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:02.823749 (+ 217us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:02.823783 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:02.823831 (+ 48us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:14:02.823869 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:02.823935 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:02.824085 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:02.783804 24980 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:02.904678 21323 rpcz_store.cc:276] 0812 19:13:59.166733 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:59.169635 (+ 2902us) service_pool.cc:227] Handling call
0812 19:14:02.238143 (+3068508us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:02.329345 (+ 91202us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:02.834339 (+504994us) spinlock_profiling.cc:243] Waited 3.07 s on lock 0x7b54002319d0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:02.834559 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7055822720}
I20260812 19:14:02.915346 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:03.052263 22052 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 4.
I20260812 19:14:03.052772 22052 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40598 (request call id 727) took 1432 ms. Trace:
I20260812 19:14:03.053293 22052 rpcz_store.cc:276] 0812 19:14:01.620088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:01.621977 (+ 1889us) service_pool.cc:227] Handling call
0812 19:14:02.915159 (+1293182us) spinlock_profiling.cc:243] Waited 4.98 ms on lock 0x7b5800005198. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:14:03.052711 (+137552us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11453184}
W20260812 19:14:02.919373 21235 rpcz_store.cc:269] 0812 19:14:00.408466 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:00.674884 (+266418us) service_pool.cc:227] Handling call
0812 19:14:02.862174 (+2187290us) spinlock_profiling.cc:243] Waited 806 ms on lock 0x7b54003bf954. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:02.862338 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1852715392}
I20260812 19:14:02.935877 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:03.054636 21328 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:14:02.935864 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:02.907101 24883 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:02.904994 24298 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:02.952378 21152 raft_consensus.cc:1261] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260812 19:14:02.962656 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 393994us
I20260812 19:14:03.063501 22055 raft_consensus.cc:1226] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 8. Current term is 9. Ops: []
I20260812 19:14:03.063866 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:03.064527 22048 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:03.065214 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:03.065829 21238 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:02.965066 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:02.964174 21329 rpcz_store.cc:276] 0812 19:14:00.328116 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:00.341999 (+ 13883us) service_pool.cc:227] Handling call
0812 19:14:00.342477 (+ 478us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:00.351397 (+ 8920us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:00.351418 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:00.355644 (+ 4226us) log.cc:842] Serialized 26 byte log entry
0812 19:14:01.776111 (+1420467us) spinlock_profiling.cc:243] Waited 31.7 ms on lock 0x7b18002b3960. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b6c345 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b89e22 00007f52d97275bb 00007f52d972d063 00007f52d972d411 00007f52d9a7a5cf
0812 19:14:01.929651 (+153540us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:14:01.929694 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:01.929769 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:02.799027 (+869258us) raft_consensus.cc:1671] finished
0812 19:14:02.799059 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:02.888556 (+ 89497us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9625,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":20535,"spinlock_wait_cycles":72921472,"thread_start_us":153322,"threads_started":1,"wal-append.queue_time_us":1636625}
W20260812 19:14:02.828749 21241 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1735) took 3559 ms (client timeout 1519 ms). Trace:
W20260812 19:14:03.067154 21241 rpcz_store.cc:269] 0812 19:13:59.269592 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:59.288508 (+ 18916us) service_pool.cc:227] Handling call
0812 19:14:02.828481 (+3539973us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b54003bea54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:02.828637 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1824644352}
I20260812 19:14:03.066382 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:02.975617 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:03.068359 21237 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
W20260812 19:14:03.068410 21233 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1620) took 2750 ms (client timeout 1645 ms). Trace:
W20260812 19:14:03.068902 21233 rpcz_store.cc:269] 0812 19:14:00.317945 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:00.590093 (+272148us) service_pool.cc:227] Handling call
0812 19:14:03.066226 (+2476133us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b5800004598. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:14:03.068338 (+ 2112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34799360}
I20260812 19:14:02.954110 24819 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:14:03.043218 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 404118us
I20260812 19:14:02.933787 24938 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:03.103691 21156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 420) took 1298 ms. Trace:
I20260812 19:14:03.104161 21156 rpcz_store.cc:276] 0812 19:14:01.805152 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:01.805467 (+ 315us) service_pool.cc:227] Handling call
0812 19:14:01.805964 (+ 497us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:01.846949 (+ 40985us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:01.846993 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:01.847569 (+ 576us) log.cc:842] Serialized 26 byte log entry
0812 19:14:02.085601 (+238032us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:14:02.085647 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:02.085719 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:03.103414 (+1017695us) raft_consensus.cc:1671] finished
0812 19:14:03.103449 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:03.103619 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":797,"raft.queue_time_us":1053,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":20224,"thread_start_us":204246,"threads_started":1,"wal-append.queue_time_us":717794}
I20260812 19:14:03.044706 22047 raft_consensus.cc:1226] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7. Ops: []
I20260812 19:14:03.109962 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:02.788457 24683 consensus_queue.cc:237] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:03.048487 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:02.961683 24821 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:14:02.950794 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 280196us
I20260812 19:14:02.918740 24787 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:03.060629 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:03.062157 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:03.062071 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:03.141865 21235 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:14:03.077255 24298 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:03.140153 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:03.078456 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:03.178751 21328 raft_consensus.cc:3037] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:14:03.179826 21241 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:14:03.080518 21237 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 8.
I20260812 19:14:03.082101 24976 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:03.083362 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:02.911851 22050 rpcz_store.cc:276] 0812 19:14:01.627325 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:01.632807 (+ 5482us) service_pool.cc:227] Handling call
0812 19:14:02.837663 (+1204856us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54004bf854. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:02.837847 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":405013760}
I20260812 19:14:03.084313 24824 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:02.947082 24847 consensus_queue.cc:260] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:14:03.085001 24674 consensus_meta.cc:287] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.104s user 0.014s sys 0.000s
W20260812 19:14:03.084998 21114 connection.cc:232] RPC call timeout handler was delayed by -2.5000261859968305s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1519s for a response
I20260812 19:14:03.085820 24965 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:03.087244 24883 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:03.045679 24844 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:03.046331 24795 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:03.090646 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:03.061225 24980 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:03.093698 21112 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:03.095633 24874 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:14:03.096663 24983 log.cc:924] Time spent T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
I20260812 19:14:03.097430 22052 raft_consensus.cc:1226] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7. Ops: []
I20260812 19:14:03.097872 22048 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:14:03.098520 24938 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:03.042680 24453 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:03.124687 21322 raft_consensus.cc:1261] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260812 19:14:03.051105 24794 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:03.047224 24606 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:03.137682 21156 raft_consensus.cc:1261] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260812 19:14:03.149801 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:03.077433 24819 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:03.056535 24812 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:03.183933 21323 raft_consensus.cc:1261] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260812 19:14:03.185323 21235 raft_consensus.cc:1261] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260812 19:14:03.186790 24298 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:03.290748 24292 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:03.381111 24841 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:03.948035 24996 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:03.339371 24838 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:03.948499 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:03.948767 24996 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:03.636515 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:03.949234 22055 raft_consensus.cc:2454] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:14:03.949868 21319 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:14:03.950286 24997 consensus_queue.cc:1046] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260812 19:14:03.951529 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:03.672940 24568 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.737s user 0.026s sys 0.017s
W20260812 19:14:03.952494 24568 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.017s user 0.027s sys 0.038s
I20260812 19:14:03.952646 24970 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:14:03.952986 21154 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.984s user 0.004s sys 0.010s
I20260812 19:14:03.952759 24812 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:14:03.953325 21154 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.984s user 0.004s sys 0.010s
I20260812 19:14:03.953660 24794 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:03.953208 24970 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:03.954919 21240 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:03.955662 21153 raft_consensus.cc:2379] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 6: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:03.955605 21154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54796 (request call id 1706) took 1378 ms. Trace:
I20260812 19:14:03.957573 24453 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:03.288033 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 148749us
I20260812 19:14:03.957257 24795 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:03.722491 24778 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.002s sys 0.009s
W20260812 19:14:03.959007 24778 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.882s user 0.002s sys 0.014s
W20260812 19:14:03.731033 21328 consensus_meta.cc:287] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.551s user 0.007s sys 0.004s
I20260812 19:14:03.960107 21328 raft_consensus.cc:2396] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260812 19:14:03.713639 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:03.822604 21149 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.149s user 0.003s sys 0.013s
W20260812 19:14:03.961603 21149 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.288s user 0.003s sys 0.015s
I20260812 19:14:03.961864 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:03.516052 21155 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.037s sys 0.043s
I20260812 19:14:03.962828 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:03.348438 21285 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.032s sys 0.078s
W20260812 19:14:03.963327 21155 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.059s user 0.044s sys 0.049s
W20260812 19:14:03.963697 21285 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.178s user 0.049s sys 0.112s
W20260812 19:14:03.673302 21150 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.781s user 0.010s sys 0.000s
W20260812 19:14:03.964731 21150 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.072s user 0.016s sys 0.000s
W20260812 19:14:03.960968 21203 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.910s user 0.003s sys 0.013s
I20260812 19:14:03.509282 24880 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:03.947638 24606 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:03.968595 24785 consensus_queue.cc:1046] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:14:03.429615 24971 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:14:03.547055 21158 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.829s user 0.016s sys 0.053s
W20260812 19:14:03.969698 21158 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.251s user 0.017s sys 0.063s
I20260812 19:14:03.969406 24971 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:03.970002 24847 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:03.968267 21329 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
W20260812 19:14:03.255354 24963 log.cc:924] Time spent T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.191s user 0.000s sys 0.000s
I20260812 19:14:03.966502 24880 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:03.803841 24988 log.cc:924] Time spent T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.289s user 0.000s sys 0.001s
I20260812 19:14:03.178725 24821 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:03.945235 21151 raft_consensus.cc:1226] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260812 19:14:03.951716 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:03.254240 24984 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:03.712795 21327 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.787s user 0.000s sys 0.010s
I20260812 19:14:03.512419 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:14:03.975581 21327 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.050s user 0.000s sys 0.015s
I20260812 19:14:03.975966 21324 raft_consensus.cc:2362] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7. Re-sending same reply.
I20260812 19:14:03.975145 21322 raft_consensus.cc:2362] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6. Re-sending same reply.
W20260812 19:14:03.700009 21321 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.762s user 0.005s sys 0.005s
W20260812 19:14:03.977257 21321 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.039s user 0.005s sys 0.010s
W20260812 19:14:03.961898 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 1.17609e+06us
I20260812 19:14:03.076475 21238 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
I20260812 19:14:03.121522 24825 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:14:03.713953 21200 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.000s sys 0.011s
I20260812 19:14:03.971525 24976 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:03.964772 24832 consensus_meta.cc:287] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 1.228s user 0.009s sys 0.005s
I20260812 19:14:03.974071 24292 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:03.974982 21236 raft_consensus.cc:2379] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 8: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
W20260812 19:14:03.141534 24927 log.cc:924] Time spent T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
W20260812 19:14:03.965916 21203 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.915s user 0.003s sys 0.014s
I20260812 19:14:03.978890 21287 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:14:04.011910 21320 raft_consensus.cc:2396] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260812 19:14:03.967796 24874 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:03.982185 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:14:03.967208 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 873522us
W20260812 19:14:03.992859 21200 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.934s user 0.003s sys 0.014s
W20260812 19:14:03.669143 21115 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.944s user 0.017s sys 0.015s
I20260812 19:14:03.956252 21154 rpcz_store.cc:276] 0812 19:14:02.576860 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:02.577001 (+ 141us) service_pool.cc:227] Handling call
0812 19:14:02.577528 (+ 527us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:02.619100 (+ 41572us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:02.619149 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:02.680461 (+ 61312us) log.cc:842] Serialized 26 byte log entry
0812 19:14:02.958867 (+278406us) raft_consensus.cc:1631] Marking committed up to 8
0812 19:14:03.953865 (+994998us) log.cc:842] Serialized 14 byte log entry
0812 19:14:03.955238 (+ 1373us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:03.955318 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:03.955366 (+ 48us) raft_consensus.cc:1671] finished
0812 19:14:03.955382 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:03.955538 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40849,"prepare.queue_time_us":993150,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":105859,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":201,"thread_start_us":1254684,"threads_started":2,"wal-append.queue_time_us":953576}
W20260812 19:14:04.003491 21204 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.942s user 0.000s sys 0.007s
W20260812 19:14:04.008430 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 59574.7us
W20260812 19:14:04.012684 21236 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57682 (request call id 351) took 3841 ms (client timeout 1853 ms). Trace:
I20260812 19:14:04.016952 21240 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
I20260812 19:14:04.020345 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:03.259609 24932 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.045997 22056 raft_consensus.cc:1226] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
I20260812 19:14:04.057523 21288 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
W20260812 19:14:04.053393 21236 rpcz_store.cc:269] 0812 19:14:00.171373 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:00.600109 (+428736us) service_pool.cc:227] Handling call
0812 19:14:03.083167 (+2483058us) spinlock_profiling.cc:243] Waited 4.96 ms on lock 0x7b5800004598. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6b52851 00007f52d6b52702 00007f52dfa36b09 00007f52de972730 00007f52dea3879b 00007f52dea38821 00007f52de9e81b0 00007f52de9e7a94 00007f52d80dbc04 00007f52d80dbb92
0812 19:14:04.012548 (+929381us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11399424}
W20260812 19:14:04.054095 21115 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.329s user 0.017s sys 0.026s
W20260812 19:14:04.059206 21204 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.998s user 0.002s sys 0.011s
W20260812 19:14:04.059418 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 1.33468e+06us
W20260812 19:14:04.059810 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 999148us
I20260812 19:14:03.999552 24844 raft_consensus.cc:3032] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 8 LEADER]: Stepping down as leader of term 8
I20260812 19:14:03.991351 24787 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.000070 24984 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:03.947636 24841 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:04.067068 21319 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:14:04.066164 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:04.065325 22048 raft_consensus.cc:2379] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 8: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:14:04.061213 21204 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:14:04.086972 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:04.087774 21204 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
W20260812 19:14:04.067888 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 120900us
I20260812 19:14:04.076756 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:04.088317 22056 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:14:04.077571 21324 raft_consensus.cc:1261] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260812 19:14:04.089905 21204 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:04.078863 21323 raft_consensus.cc:1261] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260812 19:14:03.949038 24838 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:04.082917 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 104110us
I20260812 19:14:04.095249 21287 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:04.046473 24949 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:04.096623 21287 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:04.097679 21287 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:04.095885 22051 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40598 (request call id 716) took 4903 ms. Trace:
I20260812 19:14:04.063872 25004 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:04.103816 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:04.104914 21329 raft_consensus.cc:1261] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260812 19:14:04.107587 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 47415us
I20260812 19:14:04.109711 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1848) took 3684 ms. Trace:
I20260812 19:14:04.101538 22051 rpcz_store.cc:276] 0812 19:13:59.178926 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:59.179146 (+ 220us) service_pool.cc:227] Handling call
0812 19:13:59.179753 (+ 607us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:13:59.234193 (+ 54440us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:13:59.234259 (+ 66us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:13:59.239928 (+ 5669us) log.cc:842] Serialized 26 byte log entry
0812 19:14:00.694422 (+1454494us) spinlock_profiling.cc:243] Waited 1.96 ms on lock 0x7b1800389560. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b6c345 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b89e22 00007f52d97275bb 00007f52d972d063 00007f52d972d411 00007f52d9a7a5cf
0812 19:14:01.227823 (+533401us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:14:01.251495 (+ 23672us) log.cc:842] Serialized 14 byte log entry
0812 19:14:01.451299 (+199804us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:01.451397 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:02.867763 (+1416366us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:14:04.081834 (+1214071us) raft_consensus.cc:1671] finished
0812 19:14:04.081865 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:04.082031 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43764,"prepare.queue_time_us":24277,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":140717,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":20420,"spinlock_wait_cycles":39811840,"thread_start_us":740964,"threads_started":3,"wal-append.queue_time_us":2118462}
I20260812 19:14:04.074971 24298 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260812 19:14:04.006282 24825 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:03.964877 24824 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:04.098456 22011 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:04.069674 24844 raft_consensus.cc:729] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:04.117635 22053 raft_consensus.cc:1261] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260812 19:14:03.977929 24965 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:04.110316 21327 rpcz_store.cc:276] 0812 19:14:00.425703 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:02.679320 (+2253617us) service_pool.cc:227] Handling call
0812 19:14:02.679673 (+ 353us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:02.679916 (+ 243us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:14:02.925626 (+245710us) log.cc:842] Serialized 14 byte log entry
0812 19:14:04.109205 (+1183579us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:04.109301 (+ 96us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:14:04.109368 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:04.109480 (+ 112us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:04.109633 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":587281,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":33014,"raft.queue_time_us":45751,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":11198,"thread_start_us":1428298,"threads_started":3,"wal-append.queue_time_us":1029868}
I20260812 19:14:04.082479 24832 raft_consensus.cc:2741] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260812 19:14:04.154345 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 55955.5us
I20260812 19:14:04.114821 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:04.087575 24812 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.100106 24778 raft_consensus.cc:2781] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election won for term 8
I20260812 19:14:04.086922 24787 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:04.152768 24938 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:04.140859 24880 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.153065 25028 raft_consensus.cc:2781] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:14:04.079896 25005 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:04.100274 25006 raft_consensus.cc:2726] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260812 19:14:04.158102 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 101541us
I20260812 19:14:04.115540 24821 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.092540 24984 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:04.140090 24820 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:14:04.176369 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 66725.4us
I20260812 19:14:04.140494 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:04.193610 25028 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:14:04.193873 24606 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.200479 25005 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:04.203545 24991 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:14:04.203680 24993 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:04.114228 25007 consensus_queue.cc:1046] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.095s
I20260812 19:14:04.155941 24795 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:04.176865 24787 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:04.178768 24778 raft_consensus.cc:686] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 8 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:04.208021 24841 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:04.177400 24996 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:04.208802 24292 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:04.169095 24832 raft_consensus.cc:2726] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:04.209769 24794 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:04.210263 24825 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.200747 25013 raft_consensus.cc:2726] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:14:04.119549 25004 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:04.287640 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 2086) took 1353 ms. Trace:
I20260812 19:14:04.213627 21288 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:14:04.215736 21158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 2074) took 1522 ms. Trace:
I20260812 19:14:04.232995 24938 raft_consensus.cc:2741] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260812 19:14:04.166383 25018 raft_consensus.cc:2781] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:14:04.167665 24985 raft_consensus.cc:2781] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:14:04.162345 24971 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:04.119529 24568 consensus_queue.cc:1046] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.119s
I20260812 19:14:04.260845 21115 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:04.289633 21158 rpcz_store.cc:276] 0812 19:14:02.693741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:02.717085 (+ 23344us) service_pool.cc:227] Handling call
0812 19:14:02.717515 (+ 430us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:02.717916 (+ 401us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:14:03.970374 (+1252458us) log.cc:842] Serialized 14 byte log entry
0812 19:14:04.046937 (+ 76563us) log.cc:842] Serialized 14 byte log entry
0812 19:14:04.047176 (+ 239us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:04.071051 (+ 23875us) log.cc:842] Serialized 26 byte log entry
0812 19:14:04.071196 (+ 145us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:14:04.071229 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:04.071309 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:04.202745 (+131436us) raft_consensus.cc:1671] finished
0812 19:14:04.202771 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:04.215638 (+ 12867us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32840,"prepare.queue_time_us":1233837,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":73,"raft.queue_time_us":55806,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":16971,"spinlock_wait_cycles":31741184,"thread_start_us":1298712,"threads_started":2,"wal-append.queue_time_us":170539}
I20260812 19:14:04.229527 24880 consensus_queue.cc:1035] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260812 19:14:04.259658 24820 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:04.273732 24819 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.276244 21155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 2078) took 1557 ms. Trace:
I20260812 19:14:04.115434 24949 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:04.261813 25028 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:14:04.282014 24683 raft_consensus.cc:3032] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Stepping down as leader of term 4
W20260812 19:14:04.293550 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 80031.9us
W20260812 19:14:04.295929 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 198308us
I20260812 19:14:04.287487 24874 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.301605 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:04.292563 25018 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:04.300864 21155 rpcz_store.cc:276] 0812 19:14:02.718820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:02.816407 (+ 97587us) service_pool.cc:227] Handling call
0812 19:14:02.816755 (+ 348us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:02.816993 (+ 238us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:14:03.963921 (+1146928us) log.cc:842] Serialized 14 byte log entry
0812 19:14:04.275864 (+311943us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:04.275920 (+ 56us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:14:04.275971 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:04.276040 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:04.276169 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1134862,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"raft.queue_time_us":83124,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":1371248,"threads_started":3,"wal-append.queue_time_us":240187}
I20260812 19:14:04.250072 24844 consensus_queue.cc:260] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:04.298194 25013 raft_consensus.cc:2781] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:14:04.299642 21203 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:04.312510 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
W20260812 19:14:04.316296 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 159483us
I20260812 19:14:04.332957 21327 raft_consensus.cc:1261] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260812 19:14:04.333334 22047 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:04.317929 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:04.334151 22052 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:14:04.292423 24985 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:04.263475 24990 consensus_queue.cc:1046] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260812 19:14:04.322866 25031 raft_consensus.cc:2781] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260812 19:14:04.318548 25019 log.cc:924] Time spent T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20260812 19:14:04.323369 24971 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:04.323215 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:14:04.322413 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 158480us
I20260812 19:14:04.378093 22056 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:14:04.323661 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:04.324250 25018 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:04.387737 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:04.324864 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:04.310169 21287 leader_election.cc:398] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5.
I20260812 19:14:04.335748 21288 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:14:04.304239 24838 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:04.275142 24991 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:04.309280 24683 raft_consensus.cc:729] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:04.347471 25010 raft_consensus.cc:2781] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:14:04.346006 25031 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:04.331498 24996 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:04.364179 21157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 432) took 1180 ms. Trace:
I20260812 19:14:04.414089 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:04.412933 21157 rpcz_store.cc:276] 0812 19:14:03.183771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:03.183935 (+ 164us) service_pool.cc:227] Handling call
0812 19:14:03.184296 (+ 361us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:03.289674 (+105378us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:03.289704 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:03.946329 (+656625us) log.cc:842] Serialized 26 byte log entry
0812 19:14:04.008330 (+ 62001us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:14:04.008382 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:04.363723 (+355341us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b5400194bd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:04.363777 (+ 54us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:04.363841 (+ 64us) raft_consensus.cc:1671] finished
0812 19:14:04.363854 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:04.364106 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":94490,"raft.queue_time_us":94620,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":48661,"spinlock_wait_cycles":23530752,"thread_start_us":61578,"threads_started":1,"wal-append.queue_time_us":135079}
W20260812 19:14:04.365180 24995 log.cc:924] Time spent T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.251s user 0.000s sys 0.006s
I20260812 19:14:04.369462 25016 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:04.360304 25013 raft_consensus.cc:686] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 7 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:04.371707 21236 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:14:04.433260 21236 raft_consensus.cc:595] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:14:04.374066 25004 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.275238 24993 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:04.277319 25023 consensus_queue.cc:1046] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260812 19:14:04.313884 24938 raft_consensus.cc:3032] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:14:04.388903 22055 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:14:04.391142 21327 raft_consensus.cc:1261] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260812 19:14:04.465615 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:04.389281 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:04.467355 21158 raft_consensus.cc:2379] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 7: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:14:04.468853 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:04.391549 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:04.405298 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:04.474471 21237 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:04.410513 24568 raft_consensus.cc:2781] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:04.392382 21329 raft_consensus.cc:1261] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260812 19:14:04.406803 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 106766us
I20260812 19:14:04.477551 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:04.478581 21329 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:04.480281 21329 raft_consensus.cc:1226] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5. Ops: []
I20260812 19:14:04.471863 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:04.392838 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:04.395035 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:04.408207 25049 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:04.388604 21155 raft_consensus.cc:2454] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:14:04.331097 24832 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:04.394557 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:04.486509 21318 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:04.411228 21154 raft_consensus.cc:1261] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260812 19:14:04.487212 22008 leader_election.cc:334] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.834s (SENT)
I20260812 19:14:04.487704 22008 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:04.488380 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:04.366214 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 2100) took 1019 ms. Trace:
W20260812 19:14:04.489375 22008 leader_election.cc:334] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.835s (SENT)
I20260812 19:14:04.489517 21320 rpcz_store.cc:276] 0812 19:14:03.346864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:03.510684 (+163820us) service_pool.cc:227] Handling call
0812 19:14:04.365978 (+855294us) spinlock_profiling.cc:243] Waited 48.2 ms on lock 0x7b540023e1d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:04.366142 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":110768896}
I20260812 19:14:04.414837 21235 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:04.288090 21321 rpcz_store.cc:276] 0812 19:14:02.934393 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:02.934596 (+ 203us) service_pool.cc:227] Handling call
0812 19:14:02.935098 (+ 502us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:02.935368 (+ 270us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:14:03.977679 (+1042311us) log.cc:842] Serialized 14 byte log entry
0812 19:14:04.286658 (+308979us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:04.287314 (+ 656us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:14:04.287365 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:04.287439 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:04.287573 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5907,"prepare.queue_time_us":737391,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":4249,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":1345,"thread_start_us":1341888,"threads_started":2}
I20260812 19:14:04.492695 21237 raft_consensus.cc:2454] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
W20260812 19:14:04.391861 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 56216.1us
I20260812 19:14:04.348763 24844 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:04.339718 24985 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:04.495888 21288 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:14:04.391141 22011 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:04.339018 24965 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.435055 24942 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.473136 21156 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:14:04.483705 25023 consensus_queue.cc:1225] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer 63db10063a7d4eb2a9cc9dc952654706 log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260812 19:14:04.347935 25033 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:04.498140 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:04.498361 25033 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:04.499150 22050 raft_consensus.cc:1226] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7. Ops: []
I20260812 19:14:04.499517 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:04.500340 21156 raft_consensus.cc:2379] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 7: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:14:04.500128 24993 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:04.500633 25033 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.501288 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:04.502097 21156 raft_consensus.cc:2379] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 8: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:04.423278 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:04.504889 21241 raft_consensus.cc:2354] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6.
I20260812 19:14:04.332001 24824 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.447180 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:14:04.487246 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 56491.4us
W20260812 19:14:04.506596 22008 leader_election.cc:334] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.905s (SENT)
I20260812 19:14:04.422892 21239 raft_consensus.cc:1226] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 7. Current term is 8. Ops: []
I20260812 19:14:04.391382 21287 leader_election.cc:401] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260812 19:14:04.483546 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:04.399978 24971 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:04.486565 21244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 366) took 1426 ms. Trace:
I20260812 19:14:04.416983 25031 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
W20260812 19:14:04.496883 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 105829us
I20260812 19:14:04.472425 24938 raft_consensus.cc:729] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:04.430421 24980 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.522400 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:04.421357 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:14:04.485388 21153 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51004 (request call id 415) took 2862 ms (client timeout 2307 ms). Trace:
W20260812 19:14:04.510254 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 200299us
W20260812 19:14:04.524514 21153 rpcz_store.cc:269] 0812 19:14:01.623103 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:01.623248 (+ 145us) service_pool.cc:227] Handling call
0812 19:14:04.485078 (+2861830us) spinlock_profiling.cc:243] Waited 2.32 s on lock 0x7b54001248d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:04.485287 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5327729536}
I20260812 19:14:04.525399 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:04.511263 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:14:04.515134 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 125355us
I20260812 19:14:04.387796 24874 consensus_queue.cc:1035] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.088s
I20260812 19:14:04.515700 21235 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:04.494333 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:04.514261 21244 rpcz_store.cc:276] 0812 19:14:03.059803 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:03.184648 (+124845us) service_pool.cc:227] Handling call
0812 19:14:03.979300 (+794652us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:03.979616 (+ 316us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:03.979656 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:03.979695 (+ 39us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:14:03.979747 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:03.979808 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:04.486380 (+506572us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b5400303c50. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:04.486504 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13119,"raft.queue_time_us":12333,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":19813,"spinlock_wait_cycles":1826708224}
I20260812 19:14:04.518040 21155 raft_consensus.cc:1261] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260812 19:14:04.426047 24976 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:04.518985 21115 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:04.519253 22011 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:14:04.520928 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:04.348583 24778 consensus_queue.cc:237] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:04.505539 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:04.524657 21200 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:14:04.513702 24883 raft_consensus.cc:2726] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:04.537622 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:04.537760 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:04.545724 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:14:04.517056 25021 log.cc:924] Time spent T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.166s user 0.000s sys 0.001s
I20260812 19:14:04.537024 22050 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:04.529074 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:04.540086 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:04.559383 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:04.496290 24568 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:04.538101 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:04.541651 21287 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:04.573660 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:04.574546 25027 raft_consensus.cc:2726] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:14:04.574682 24568 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:04.447306 25016 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:04.576363 25029 raft_consensus.cc:2781] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:14:04.576864 25029 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:04.577231 25029 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:04.578083 25016 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:04.578989 25013 consensus_queue.cc:237] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:04.579941 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:04.580631 21235 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:14:04.581562 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:04.517082 24837 raft_consensus.cc:2781] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:14:04.516410 24984 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:04.582791 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:04.573683 21239 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:04.584112 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:04.561369 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:04.569818 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:04.514223 24982 consensus_queue.cc:1035] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
I20260812 19:14:04.582537 24935 raft_consensus.cc:2781] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader election won for term 8
I20260812 19:14:04.513281 21329 raft_consensus.cc:3032] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:14:04.571560 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:04.595708 21329 raft_consensus.cc:729] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:04.596486 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:04.597321 21237 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:14:04.517022 24838 raft_consensus.cc:2726] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5.
I20260812 19:14:04.585227 21241 raft_consensus.cc:2362] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6. Re-sending same reply.
I20260812 19:14:04.599138 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:04.599771 21239 raft_consensus.cc:3037] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
W20260812 19:14:04.570585 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 46005.9us
I20260812 19:14:04.589601 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:04.591243 25005 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.431802 24996 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:14:04.594506 21114 leader_election.cc:398] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6.
I20260812 19:14:04.547686 21324 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:04.516950 24942 raft_consensus.cc:2781] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:14:04.598280 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:04.599146 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:04.602391 21287 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
W20260812 19:14:04.585825 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 66913.5us
I20260812 19:14:04.602286 21114 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:14:04.604996 24984 raft_consensus.cc:2726] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 19:14:04.582845 24837 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:04.605593 21149 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 2072) took 2017 ms. Trace:
I20260812 19:14:04.605928 24837 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:04.606453 21241 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:14:04.607209 21241 raft_consensus.cc:595] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:14:04.607576 21241 raft_consensus.cc:603] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:14:04.608526 24996 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:04.514797 24832 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:04.611855 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:04.521693 24821 consensus_queue.cc:1035] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260812 19:14:04.524715 25023 consensus_queue.cc:1046] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.074s
W20260812 19:14:04.617823 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 45595.2us
I20260812 19:14:04.602939 21236 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 8.
I20260812 19:14:04.596997 21329 consensus_queue.cc:260] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:04.735743 21329 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:04.602844 21244 raft_consensus.cc:2362] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 9. Re-sending same reply.
I20260812 19:14:04.734584 24787 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.603690 21287 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:04.606073 21149 rpcz_store.cc:276] 0812 19:14:02.588617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:02.672374 (+ 83757us) service_pool.cc:227] Handling call
0812 19:14:02.672722 (+ 348us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:03.962257 (+1289535us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:03.962287 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:03.993305 (+ 31018us) log.cc:842] Serialized 26 byte log entry
0812 19:14:04.124245 (+130940us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:14:04.124283 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:04.124365 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:04.605340 (+480975us) raft_consensus.cc:1671] finished
0812 19:14:04.605364 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:04.605522 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":76030,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":10787,"thread_start_us":1355450,"threads_started":2,"wal-append.queue_time_us":283491}
I20260812 19:14:04.611461 24942 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
W20260812 19:14:04.680212 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 64829.3us
I20260812 19:14:04.610859 24935 raft_consensus.cc:686] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:14:04.611199 25050 raft_consensus.cc:2781] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 19:14:04.699489 25054 consensus_queue.cc:1035] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20260812 19:14:04.505615 25049 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:04.699229 24971 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:04.749346 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:04.656008 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 52617.3us
I20260812 19:14:04.694712 24989 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:04.752003 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:04.540191 24683 consensus_queue.cc:260] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:04.657799 24795 consensus_queue.cc:1035] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260812 19:14:04.653575 21150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 2076) took 1935 ms. Trace:
I20260812 19:14:04.708154 25028 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:04.691938 24985 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:04.709929 24991 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:04.711624 25018 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:04.717044 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 114838us
W20260812 19:14:04.717836 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 99060.3us
I20260812 19:14:04.726344 24568 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:04.677448 25044 consensus_queue.cc:1035] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20260812 19:14:04.725036 25033 raft_consensus.cc:2781] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:14:04.732589 25029 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:04.549628 24681 consensus_queue.cc:1035] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
W20260812 19:14:04.740139 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 136516us
I20260812 19:14:04.824086 21287 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:04.738435 22047 raft_consensus.cc:2454] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:14:04.820057 24874 consensus_queue.cc:1035] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260812 19:14:04.747386 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:04.826175 21320 raft_consensus.cc:2379] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 7: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:04.826530 21157 raft_consensus.cc:1261] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260812 19:14:04.826823 22051 raft_consensus.cc:1226] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6. Ops: []
W20260812 19:14:04.761030 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 160707us
I20260812 19:14:04.748464 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:04.827859 21327 raft_consensus.cc:2379] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 7: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:14:04.748638 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:04.749282 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 10 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:04.592993 24883 consensus_queue.cc:1035] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.071s
I20260812 19:14:04.770598 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:04.770648 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:04.768994 21329 raft_consensus.cc:1261] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260812 19:14:04.756145 21239 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:04.829921 21329 pending_rounds.cc:76] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412: Aborting all ops after (but not including) 4
I20260812 19:14:04.830219 21329 pending_rounds.cc:97] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260812 19:14:04.773721 21150 rpcz_store.cc:276] 0812 19:14:02.717967 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:02.790929 (+ 72962us) service_pool.cc:227] Handling call
0812 19:14:02.815561 (+ 24632us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:02.815918 (+ 357us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:02.815957 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:02.817426 (+ 1469us) log.cc:842] Serialized 26 byte log entry
0812 19:14:03.965768 (+1148342us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:14:03.966355 (+ 587us) log.cc:842] Serialized 14 byte log entry
0812 19:14:03.966798 (+ 443us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:03.966875 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:04.364448 (+397573us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:14:04.613894 (+249446us) raft_consensus.cc:1671] finished
0812 19:14:04.653298 (+ 39404us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:04.653496 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":25886,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":45,"raft.queue_time_us":349,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":115,"spinlock_wait_cycles":53176576,"thread_start_us":1073291,"threads_started":1,"wal-append.queue_time_us":1154463}
I20260812 19:14:04.778021 21236 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:14:04.830781 21237 raft_consensus.cc:1261] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260812 19:14:04.831148 21236 raft_consensus.cc:595] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:14:04.750095 25025 raft_consensus.cc:979] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa: : Instructing follower 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:14:04.836722 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:04.788802 21114 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:04.791530 22050 raft_consensus.cc:1226] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:14:04.740865 24942 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:14:04.740633 25023 raft_consensus.cc:2726] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260812 19:14:04.792554 21112 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:14:04.794209 25036 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:04.745349 25010 raft_consensus.cc:686] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:04.852227 25010 consensus_queue.cc:237] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:04.750553 25050 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:04.854867 25050 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:04.855260 21322 raft_consensus.cc:2396] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260812 19:14:04.855665 24832 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:04.856984 25031 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:04.857843 25029 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.858352 21158 raft_consensus.cc:2354] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7.
I20260812 19:14:04.866302 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51552 (request call id 2107) took 1018 ms. Trace:
I20260812 19:14:04.893860 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:04.894054 21200 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:04.808981 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:04.830255 21321 raft_consensus.cc:2396] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260812 19:14:04.894229 21319 rpcz_store.cc:276] 0812 19:14:03.851168 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:03.951243 (+100075us) service_pool.cc:227] Handling call
0812 19:14:04.866053 (+914810us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b54000613d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:04.866205 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":263766784}
I20260812 19:14:04.830552 21329 raft_consensus.cc:2866] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:14:04.783207 21203 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:04.832508 21115 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:04.895844 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:14:04.896206 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 63811us
W20260812 19:14:04.743203 24847 consensus_meta.cc:287] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.773s user 0.014s sys 0.000s
I20260812 19:14:04.756661 24837 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:04.902271 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 119138us
I20260812 19:14:04.902869 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:04.903447 21203 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:04.829339 21324 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:14:04.844471 21318 raft_consensus.cc:2379] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 8: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:04.844901 21320 raft_consensus.cc:3032] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 8 LEADER]: Stepping down as leader of term 8
I20260812 19:14:04.845480 21239 raft_consensus.cc:1261] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260812 19:14:04.847615 21238 raft_consensus.cc:1226] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:14:04.750288 24935 consensus_queue.cc:237] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:04.773814 24989 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:04.858768 24996 raft_consensus.cc:2726] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260812 19:14:04.807286 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:04.782893 24683 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:14:04.860495 21236 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:04.861570 25031 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:04.861934 24970 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:04.952538 21244 raft_consensus.cc:1226] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:14:04.872144 25018 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:04.874720 24985 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:04.876433 25028 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:04.877825 25049 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:04.883260 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 59301.6us
I20260812 19:14:04.885990 24949 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:04.887001 25036 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:04.889057 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
W20260812 19:14:04.889792 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 97398.3us
I20260812 19:14:04.890682 25039 raft_consensus.cc:2726] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:14:04.890956 25054 raft_consensus.cc:1067] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Signalling peer 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:14:04.891449 24681 raft_consensus.cc:2726] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260812 19:14:04.891791 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 103139us
I20260812 19:14:04.894655 22047 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:14:04.827812 21204 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:04.980159 21114 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:04.836959 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:04.896798 21329 raft_consensus.cc:1261] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260812 19:14:04.981099 21327 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
W20260812 19:14:04.981321 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 153598us
I20260812 19:14:04.982642 21204 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:04.984352 21204 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:14:04.899255 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:04.985715 22048 raft_consensus.cc:2379] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 7: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:14:04.907186 21320 raft_consensus.cc:729] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:04.909817 21200 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:04.913250 21203 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:04.916354 21319 raft_consensus.cc:1226] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 7 LEADER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
I20260812 19:14:04.908442 21243 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:14:04.826299 25033 raft_consensus.cc:686] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:04.944643 22053 raft_consensus.cc:1226] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7. Ops: []
I20260812 19:14:04.908723 24991 raft_consensus.cc:2781] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:14:04.911875 24812 raft_consensus.cc:2726] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:14:04.912119 24819 raft_consensus.cc:2726] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:04.941924 25006 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260812 19:14:04.989223 22008 leader_election.cc:334] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.066s (SENT)
I20260812 19:14:04.989746 25018 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:14:04.993229 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 83534.8us
I20260812 19:14:04.994956 21204 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:04.996138 21114 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:14:04.987308 21320 consensus_queue.cc:260] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:14:05.003788 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 57213.4us
W20260812 19:14:05.005985 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 92828.8us
I20260812 19:14:04.991817 21287 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:05.013815 21320 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:05.014915 21287 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:05.012468 21200 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:05.015272 21236 raft_consensus.cc:1261] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260812 19:14:04.981418 24997 raft_consensus.cc:3032] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:14:04.948649 24841 consensus_queue.cc:1046] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260812 19:14:04.998718 24843 raft_consensus.cc:2726] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:04.997751 24844 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:05.029353 21236 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:14:05.003952 25031 raft_consensus.cc:2726] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:14:05.042986 21236 raft_consensus.cc:595] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:14:05.043630 21236 raft_consensus.cc:603] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:14:05.045223 21320 raft_consensus.cc:1261] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260812 19:14:05.045709 21320 pending_rounds.cc:76] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412: Aborting all ops after (but not including) 5
W20260812 19:14:05.041795 22008 leader_election.cc:334] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.165s (SENT)
I20260812 19:14:05.046301 22008 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:05.007910 24606 raft_consensus.cc:2726] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:14:05.047619 21320 pending_rounds.cc:97] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260812 19:14:05.006470 25028 raft_consensus.cc:2726] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:05.048471 21320 raft_consensus.cc:2866] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:14:05.023195 21287 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:05.009936 24991 raft_consensus.cc:686] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:05.065197 24991 consensus_queue.cc:237] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.011229 25006 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:05.066917 25006 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
W20260812 19:14:05.067612 25067 log.cc:924] Time spent T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.175s user 0.000s sys 0.001s
I20260812 19:14:05.071334 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:05.073310 24837 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:05.005327 24821 consensus_queue.cc:1046] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
I20260812 19:14:05.075191 25023 raft_consensus.cc:2726] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260812 19:14:05.047580 24832 raft_consensus.cc:2781] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:14:05.077350 24832 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:05.077769 24832 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:05.050134 19738 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260812 19:14:05.055056 24824 raft_consensus.cc:2781] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:14:05.022397 24996 raft_consensus.cc:2781] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:14:05.057291 21287 leader_election.cc:398] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7.
I20260812 19:14:05.019686 24997 raft_consensus.cc:729] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:05.095420 21287 leader_election.cc:401] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260812 19:14:05.010200 24949 consensus_queue.cc:1046] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
W20260812 19:14:05.095844 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 99827.5us
I20260812 19:14:05.108286 21156 raft_consensus.cc:1261] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260812 19:14:05.054747 24985 raft_consensus.cc:2726] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:14:05.086393 24568 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:05.024179 25018 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.178114 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:05.179737 21150 raft_consensus.cc:2396] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260812 19:14:05.182031 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:05.095162 24996 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:05.159092 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:05.169688 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:05.117609 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:05.046571 25013 consensus_queue.cc:1035] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.071s
I20260812 19:14:05.096311 24824 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:05.111163 21152 raft_consensus.cc:1226] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260812 19:14:05.199854 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:05.211377 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:05.102291 25023 raft_consensus.cc:2726] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7.
I20260812 19:14:05.052369 24844 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:05.134523 25033 consensus_queue.cc:237] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.176786 25044 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:05.113916 24841 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:05.235088 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:05.221483 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:05.243348 24841 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.170164 25031 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:14:05.246045 25076 log.cc:924] Time spent T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.153s user 0.001s sys 0.000s
I20260812 19:14:05.223382 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:05.221483 24824 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:05.222599 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:05.268644 21152 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8.
I20260812 19:14:05.257375 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:05.207461 24996 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:05.212575 21155 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
W20260812 19:14:05.271927 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 99046.1us
I20260812 19:14:05.273432 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:05.278004 25023 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:05.241923 24989 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:05.280584 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:05.281226 24838 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
W20260812 19:14:05.286567 21287 connection.cc:232] RPC call timeout handler was delayed by -1.8025812751147896s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
W20260812 19:14:05.287086 21287 connection.cc:232] RPC call timeout handler was delayed by -1.7638812750810757s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15600000000000003s for a response
W20260812 19:14:05.287433 21287 connection.cc:232] RPC call timeout handler was delayed by -1.75758127507288s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1567s for a response
W20260812 19:14:05.287762 21287 connection.cc:232] RPC call timeout handler was delayed by -1.6891812750836834s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1643s for a response
I20260812 19:14:05.278739 25023 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.237115 24942 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:05.288057 21287 connection.cc:232] RPC call timeout handler was delayed by -1.6018812750698999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17400000000000002s for a response
I20260812 19:14:05.288702 24838 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.289875 24298 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:05.227720 24837 consensus_queue.cc:1046] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260812 19:14:05.270596 24832 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:14:05.295490 22011 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.003s sys 0.007s
I20260812 19:14:05.302891 21239 raft_consensus.cc:1261] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260812 19:14:05.241252 25050 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:05.304768 21287 connection.cc:232] RPC call timeout handler was delayed by -1.4848812751006335s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18700000000000003s for a response
I20260812 19:14:05.308104 22056 raft_consensus.cc:1226] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5. Ops: []
I20260812 19:14:05.309742 24298 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Advancing to term 10
I20260812 19:14:05.317588 24820 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:05.313629 21287 connection.cc:232] RPC call timeout handler was delayed by -1.4155812750104815s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1947s for a response
W20260812 19:14:05.321929 21287 connection.cc:232] RPC call timeout handler was delayed by -1.3867812750395387s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19790000000000002s for a response
W20260812 19:14:05.322326 21287 connection.cc:232] RPC call timeout handler was delayed by -1.2310812750365585s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21520000000000003s for a response
I20260812 19:14:05.263746 25044 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:05.269580 25031 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:05.331077 24877 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:05.333050 25031 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:05.341176 25056 raft_consensus.cc:2726] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:14:05.346908 25005 raft_consensus.cc:2781] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:14:05.347555 25029 raft_consensus.cc:2726] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:05.347697 25005 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:05.348076 24821 raft_consensus.cc:3032] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:14:05.324932 21153 raft_consensus.cc:1226] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 5. Current term is 6. Ops: []
W20260812 19:14:05.342046 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 329638us
I20260812 19:14:05.346972 25054 raft_consensus.cc:2781] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260812 19:14:05.341751 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 346900us
I20260812 19:14:05.346478 24824 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.351850 25054 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:05.352301 21204 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:14:05.325350 24837 raft_consensus.cc:3032] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:14:05.353103 24837 raft_consensus.cc:729] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:05.338315 24832 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:05.355099 24918 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260812 19:14:05.348145 25005 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:05.322753 24942 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:05.354094 24837 consensus_queue.cc:260] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.357600 24837 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:05.294597 24938 consensus_queue.cc:260] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.301638 25023 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:05.352377 25054 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
W20260812 19:14:05.360782 25060 log.cc:924] Time spent T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.355s user 0.002s sys 0.000s
I20260812 19:14:05.362486 24938 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:05.307984 24949 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:05.363330 24949 raft_consensus.cc:504] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.365356 25036 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:05.365654 24949 leader_election.cc:288] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:05.367120 21204 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:05.348744 24821 raft_consensus.cc:729] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:05.368054 24989 consensus_queue.cc:1035] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260812 19:14:05.368467 24821 consensus_queue.cc:260] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.369614 25028 raft_consensus.cc:2726] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:14:05.369822 21200 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:05.371232 21200 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:05.371340 24606 raft_consensus.cc:2781] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:05.371846 24606 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:05.372301 24606 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:05.373098 24843 raft_consensus.cc:1050] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: attempting to promote NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 to VOTER
I20260812 19:14:05.373386 21200 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:05.375046 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.375804 21244 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:14:05.375509 24843 consensus_queue.cc:237] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 5, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.376694 24785 raft_consensus.cc:2781] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:05.377140 24785 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:05.377311 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.377568 24785 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:05.378602 24819 raft_consensus.cc:979] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa: : Instructing follower 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:14:05.379161 24819 raft_consensus.cc:1067] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Signalling peer 93578836968649f6bfcaecc709871412 to start an election
I20260812 19:14:05.385556 24971 consensus_queue.cc:1046] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260812 19:14:05.386965 24841 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:05.387082 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:05.388737 24997 consensus_queue.cc:260] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:14:05.389443 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.390326 24997 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:05.323961 21287 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:05.410085 21233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1739) took 1006 ms. Trace:
I20260812 19:14:05.422073 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:05.393175 25018 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:05.402357 21154 raft_consensus.cc:2454] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 4.
I20260812 19:14:05.432355 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:05.407004 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:05.432610 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.429414 21233 rpcz_store.cc:276] 0812 19:14:04.403356 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:04.403623 (+ 267us) service_pool.cc:227] Handling call
0812 19:14:04.433683 (+ 30060us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:04.571294 (+137611us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:04.571346 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:04.596213 (+ 24867us) log.cc:842] Serialized 26 byte log entry
0812 19:14:04.783763 (+187550us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:14:04.999473 (+215710us) log.cc:842] Serialized 14 byte log entry
0812 19:14:05.340378 (+340905us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b18003e3c80. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b014ca 00007f52e01d6f99 00007f52d69c5990 00007f52d6b87864 00007f52d6b87155 00007f52d9a9e3d0 00007f52d9ab0576 00007f52d9ac10d7 00007f52d9ac1058
0812 19:14:05.340584 (+ 206us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:05.340688 (+ 104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:05.360068 (+ 19380us) spinlock_profiling.cc:243] Waited 31.7 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:14:05.409794 (+ 49726us) raft_consensus.cc:1671] finished
0812 19:14:05.409818 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:05.410003 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":459874,"prepare.queue_time_us":174328,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":421874,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":5816,"spinlock_wait_cycles":76066688,"thread_start_us":362705,"threads_started":3,"wal-append.queue_time_us":287055}
I20260812 19:14:05.440343 21320 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b"
dest_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:51552
I20260812 19:14:05.333333 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:05.441053 21320 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:14:05.441543 21320 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:05.442147 22055 raft_consensus.cc:2379] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 7: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:14:05.400185 24989 consensus_queue.cc:1035] T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 19:14:05.399590 25050 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:05.421377 22011 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.850s user 0.004s sys 0.008s
W20260812 19:14:05.447067 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 899222us
W20260812 19:14:05.443296 22008 leader_election.cc:334] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.168s (SENT)
I20260812 19:14:05.408211 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:14:05.402096 21240 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57682 (request call id 356) took 2432 ms (client timeout 2836 ms). Trace:
I20260812 19:14:05.447988 22008 leader_election.cc:302] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:05.446213 22051 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:14:05.448411 21240 rpcz_store.cc:269] 0812 19:14:02.969996 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:03.046569 (+ 76573us) service_pool.cc:227] Handling call
0812 19:14:05.401861 (+2355292us) spinlock_profiling.cc:243] Waited 844 ms on lock 0x7b5400303254. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:05.402010 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1941042048}
I20260812 19:14:05.449712 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.450469 21240 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:14:05.397557 24825 raft_consensus.cc:3032] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:14:05.397145 25006 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:05.404084 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 395501us
I20260812 19:14:05.390798 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.406674 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.409164 21321 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db"
dest_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:51552
I20260812 19:14:05.431320 25023 raft_consensus.cc:2726] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260812 19:14:05.369846 24821 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:05.455261 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:05.474206 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.474677 21112 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:05.475354 21322 raft_consensus.cc:2454] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 9.
I20260812 19:14:05.460824 21321 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 19:14:05.476740 21321 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:05.476807 22053 raft_consensus.cc:1226] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:14:05.476825 24976 consensus_queue.cc:1035] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260812 19:14:05.455562 24942 raft_consensus.cc:2726] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:14:05.481515 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 2111) took 1148 ms. Trace:
I20260812 19:14:05.432952 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:05.481999 21328 rpcz_store.cc:276] 0812 19:14:04.332820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:04.334216 (+ 1396us) service_pool.cc:227] Handling call
0812 19:14:04.359768 (+ 25552us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:04.781828 (+422060us) raft_consensus.cc:1520] Early marking committed up to index 3
0812 19:14:04.798229 (+ 16401us) log.cc:842] Serialized 14 byte log entry
0812 19:14:04.934758 (+136529us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:04.934848 (+ 90us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:14:04.934927 (+ 79us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:05.481270 (+546343us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b540039a8d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:05.481320 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:05.481451 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11538,"prepare.queue_time_us":16184,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":68,"raft.queue_time_us":833,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":10165,"spinlock_wait_cycles":970116480,"thread_start_us":123290,"threads_started":1,"wal-append.queue_time_us":245440}
I20260812 19:14:05.455583 24683 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:05.484163 24683 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:05.484644 24844 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:05.491362 21151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54796 (request call id 1705) took 2920 ms. Trace:
I20260812 19:14:05.491706 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:05.491916 21151 rpcz_store.cc:276] 0812 19:14:02.570984 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:02.574507 (+ 3523us) service_pool.cc:227] Handling call
0812 19:14:02.574872 (+ 365us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:03.974447 (+1399575us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:05.491068 (+1516621us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b540006ead4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:05.491293 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3076896000}
I20260812 19:14:05.492518 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.401978 25044 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:05.473662 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:05.468010 21320 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:05.494151 22048 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:05.336942 24838 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:05.492619 24683 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:05.500878 21320 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:05.501430 21241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1753) took 1080 ms. Trace:
I20260812 19:14:05.502415 21321 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.450539 22011 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:14:05.474094 21236 raft_consensus.cc:1226] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 9 LEADER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 8. Current term is 9. Ops: []
I20260812 19:14:05.497195 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:14:05.497365 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 173465us
I20260812 19:14:05.505255 21244 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8.
W20260812 19:14:05.504400 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 53971.3us
I20260812 19:14:05.505448 21287 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:05.498214 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:05.506672 22011 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:14:05.509622 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.510185 21287 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:05.512346 24877 raft_consensus.cc:2781] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Leader election won for term 9
I20260812 19:14:05.512187 24453 consensus_queue.cc:1035] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:14:05.513177 24877 raft_consensus.cc:686] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [term 9 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:14:05.514199 24877 consensus_queue.cc:237] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.514211 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:05.516139 22053 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:05.508126 25010 raft_consensus.cc:2726] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260812 19:14:05.518273 21235 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1754) took 1081 ms. Trace:
I20260812 19:14:05.456465 24825 raft_consensus.cc:729] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:05.518767 21235 rpcz_store.cc:276] 0812 19:14:04.437533 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:04.615299 (+177766us) service_pool.cc:227] Handling call
0812 19:14:04.699611 (+ 84312us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:04.700374 (+ 763us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:04.700405 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:04.701611 (+ 1206us) log.cc:842] Serialized 26 byte log entry
0812 19:14:04.757129 (+ 55518us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:14:05.368812 (+611683us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x7b180028a4a0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d973582c 00007f52d6b6c9dc 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b87155 00007f52dd47b167 00007f52d9ab0807 00007f52d9abff0f
0812 19:14:05.368974 (+ 162us) log.cc:842] Serialized 14 byte log entry
0812 19:14:05.369199 (+ 225us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:05.369273 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:05.455623 (+ 86350us) spinlock_profiling.cc:243] Waited 28.7 ms on lock 0x7b180028a4a0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a638 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
0812 19:14:05.518021 (+ 62398us) raft_consensus.cc:1671] finished
0812 19:14:05.518045 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:05.518211 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":121264,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":15341,"raft.queue_time_us":1310,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":197,"spinlock_wait_cycles":68568832,"thread_start_us":233886,"threads_started":2,"wal-append.queue_time_us":141948}
W20260812 19:14:05.520421 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 45869.5us
I20260812 19:14:05.520267 22051 raft_consensus.cc:2454] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:14:05.524363 21112 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:05.521519 22048 raft_consensus.cc:2454] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 9.
I20260812 19:14:05.525419 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.525681 21288 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:14:05.527019 21114 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:14:05.527021 22047 raft_consensus.cc:1226] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:14:05.531015 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.535650 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.536114 21115 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:05.436897 24606 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:05.398483 24996 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.539407 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.548249 21112 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:05.548821 21243 raft_consensus.cc:2454] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:14:05.542179 25006 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:05.542486 24292 raft_consensus.cc:3032] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:14:05.537190 24971 raft_consensus.cc:2781] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:14:05.520390 25023 raft_consensus.cc:2726] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:14:05.499698 24820 consensus_queue.cc:1035] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260812 19:14:05.544178 24824 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:05.545262 25049 raft_consensus.cc:2781] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader election won for term 9
I20260812 19:14:05.606032 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.537521 25085 raft_consensus.cc:2726] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:14:05.617012 21149 raft_consensus.cc:1226] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7. Ops: []
I20260812 19:14:05.519183 24942 raft_consensus.cc:2781] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:14:05.523161 24838 raft_consensus.cc:2741] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260812 19:14:05.549790 21112 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:05.505718 21321 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:05.624442 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 74744.1us
I20260812 19:14:05.522181 21287 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
W20260812 19:14:05.625602 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 103511us
I20260812 19:14:05.626366 21287 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:14:05.559352 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 52754.3us
I20260812 19:14:05.627758 22011 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:05.628191 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:05.629096 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:05.629607 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 10 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:05.629773 21320 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:14:05.630100 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:05.569283 22053 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8.
W20260812 19:14:05.579075 21318 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.005s sys 0.035s
I20260812 19:14:05.533246 24990 raft_consensus.cc:2781] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:05.593570 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.638103 24990 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:05.638345 25005 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.623978 25049 raft_consensus.cc:686] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 9 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:05.633417 24844 raft_consensus.cc:2726] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:14:05.637658 25085 raft_consensus.cc:2781] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:05.600912 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:05.539256 24991 raft_consensus.cc:2781] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 19:14:05.576892 24292 raft_consensus.cc:729] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:05.508118 25007 consensus_queue.cc:1046] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.069s
I20260812 19:14:05.439253 24298 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:14:05.527917 24683 consensus_queue.cc:1035] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260812 19:14:05.501830 21241 rpcz_store.cc:276] 0812 19:14:04.420962 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:04.608270 (+187308us) service_pool.cc:227] Handling call
0812 19:14:04.608626 (+ 356us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:04.609016 (+ 390us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:04.609051 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:04.609550 (+ 499us) log.cc:842] Serialized 26 byte log entry
0812 19:14:04.772481 (+162931us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:14:05.371376 (+598895us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b180043b520. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b014ca 00007f52e01d6f99 00007f52dd94ebcf 00007f52d6b8a7b6 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d
0812 19:14:05.478115 (+106739us) spinlock_profiling.cc:243] Waited 3.05 ms on lock 0x7b180043b520. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d973582c 00007f52d6b6c9dc 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b87155 00007f52dd47b167 00007f52d9ab0807 00007f52d9abff0f
0812 19:14:05.478286 (+ 171us) log.cc:842] Serialized 14 byte log entry
0812 19:14:05.501109 (+ 22823us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:05.501192 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:05.501236 (+ 44us) raft_consensus.cc:1671] finished
0812 19:14:05.501245 (+ 9us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:05.501370 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46292,"prepare.queue_time_us":129217,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":28871,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":40342,"spinlock_wait_cycles":30313088,"thread_start_us":309791,"threads_started":2,"wal-append.queue_time_us":198812}
W20260812 19:14:05.630856 21318 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s user 0.006s sys 0.035s
I20260812 19:14:05.631042 21319 raft_consensus.cc:1226] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 6. Current term is 7. Ops: []
I20260812 19:14:05.631565 22050 raft_consensus.cc:1226] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 7. Current term is 8. Ops: []
I20260812 19:14:05.631570 21288 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:14:05.633761 24980 raft_consensus.cc:2781] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:14:05.633502 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.635443 24942 raft_consensus.cc:686] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:14:05.635293 25033 consensus_queue.cc:1035] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:14:05.636162 24832 raft_consensus.cc:2781] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:14:05.636482 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:05.652124 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1768) took 1147 ms. Trace:
I20260812 19:14:05.605799 25018 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:05.637156 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:05.637720 24996 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:05.759783 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:05.602741 22046 raft_consensus.cc:1226] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5. Ops: []
I20260812 19:14:05.577467 24841 raft_consensus.cc:2781] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260812 19:14:05.650521 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 636394us
I20260812 19:14:05.681734 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:05.639627 21238 raft_consensus.cc:3037] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:05.762882 21203 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:05.764628 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:05.765512 22051 raft_consensus.cc:2354] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7.
I20260812 19:14:05.687885 21243 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8.
W20260812 19:14:05.709668 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 173616us
I20260812 19:14:05.745736 21239 rpcz_store.cc:276] 0812 19:14:04.504466 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:04.950308 (+445842us) service_pool.cc:227] Handling call
0812 19:14:05.302469 (+352161us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:05.312120 (+ 9651us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:05.651853 (+339733us) spinlock_profiling.cc:243] Waited 330 ms on lock 0x7b5400303750. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:05.652038 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":757923072}
I20260812 19:14:05.769001 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.770092 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:05.771253 21238 raft_consensus.cc:2454] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:05.662266 24606 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:05.576779 25044 consensus_queue.cc:1046] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
W20260812 19:14:05.754508 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 128211us
W20260812 19:14:05.687827 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 56347.8us
I20260812 19:14:05.758409 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:05.785624 21203 leader_election.cc:302] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:05.617952 24971 raft_consensus.cc:686] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 6 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:05.789919 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:05.647640 25005 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:05.791755 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.822196 21243 raft_consensus.cc:2362] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7. Re-sending same reply.
I20260812 19:14:05.794457 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:05.824954 21235 raft_consensus.cc:1799] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Handling vote request from an unknown peer 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:14:05.825536 21235 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
I20260812 19:14:05.657968 24298 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 10 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:05.803050 21287 leader_election.cc:398] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 8. Current term is 9.
I20260812 19:14:05.830998 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:05.781262 24841 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:05.758567 24838 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:05.802982 21203 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:05.792061 24606 raft_consensus.cc:2726] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:14:05.651746 24292 consensus_queue.cc:260] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.805326 24795 consensus_queue.cc:1046] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260812 19:14:05.778990 25018 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:05.807035 25045 raft_consensus.cc:2726] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:14:05.807525 24918 consensus_queue.cc:1035] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 19:14:05.746816 24844 raft_consensus.cc:2781] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader election won for term 8
I20260812 19:14:05.647503 25085 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:05.638603 24990 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:05.650970 24991 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:05.745334 24832 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:05.638880 24785 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:05.843290 24841 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:05.640066 25049 consensus_queue.cc:237] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.659226 24825 consensus_queue.cc:260] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.868837 25005 raft_consensus.cc:2726] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:14:05.855496 24838 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:05.883841 24996 consensus_queue.cc:1159] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "63db10063a7d4eb2a9cc9dc952654706" responder_term: 5 status { last_received { term: 4 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260812 19:14:05.890686 25085 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:05.882790 24942 consensus_queue.cc:237] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.897221 24991 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Advancing to term 9
W20260812 19:14:05.910336 25090 log.cc:924] Time spent T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20260812 19:14:05.921576 25018 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:14:05.923465 25054 consensus_meta.cc:287] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.563s user 0.010s sys 0.002s
I20260812 19:14:05.923987 25054 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:05.925088 25085 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:05.926421 25054 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:05.927359 25085 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:05.816766 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 281223us
I20260812 19:14:05.968726 21114 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:14:05.810650 21204 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:14:05.979526 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:05.981479 21158 raft_consensus.cc:2379] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 7: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:14:05.981364 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 170667us
W20260812 19:14:05.865581 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 62650.2us
W20260812 19:14:05.848533 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 45608.1us
I20260812 19:14:05.828274 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:05.987191 21327 raft_consensus.cc:2454] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:14:05.988252 24980 raft_consensus.cc:686] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 7 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:14:05.853309 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:05.989681 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:05.808460 21237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1749) took 1389 ms. Trace:
I20260812 19:14:05.847498 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:05.991659 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:05.992624 24453 consensus_queue.cc:1046] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260812 19:14:05.855737 21323 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
W20260812 19:14:05.905829 24837 consensus_meta.cc:287] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.548s user 0.011s sys 0.000s
I20260812 19:14:05.810290 21321 raft_consensus.cc:1226] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 7. Current term is 8. Ops: []
I20260812 19:14:05.883170 24292 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
W20260812 19:14:05.997139 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 122202us
W20260812 19:14:05.997486 25103 log.cc:924] Time spent T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20260812 19:14:05.977053 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:05.991453 24980 consensus_queue.cc:237] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:05.991199 21237 rpcz_store.cc:276] 0812 19:14:04.418872 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:04.602251 (+183379us) service_pool.cc:227] Handling call
0812 19:14:04.830398 (+228147us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:04.831180 (+ 782us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:05.808168 (+976988us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b54000b5250. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:05.808378 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":523907328}
I20260812 19:14:06.005606 24838 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:06.016274 24674 raft_consensus.cc:2781] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:14:06.019371 25054 raft_consensus.cc:2726] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:14:05.899245 24832 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:05.894994 24971 consensus_queue.cc:237] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:06.041175 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 65708.4us
I20260812 19:14:06.043074 24785 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:05.894287 24844 raft_consensus.cc:686] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 8 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:06.040335 24674 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:06.058254 24674 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:06.053649 21327 raft_consensus.cc:2454] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 9.
I20260812 19:14:06.054816 22011 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:06.062784 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:06.074723 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:14:06.080385 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 66299.6us
I20260812 19:14:06.087975 24844 consensus_queue.cc:237] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:06.088932 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 2128) took 1195 ms. Trace:
I20260812 19:14:06.089383 21318 rpcz_store.cc:276] 0812 19:14:04.893882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:04.906436 (+ 12554us) service_pool.cc:227] Handling call
0812 19:14:04.906811 (+ 375us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:04.943330 (+ 36519us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:04.943372 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:04.943978 (+ 606us) log.cc:842] Serialized 26 byte log entry
0812 19:14:05.385160 (+441182us) spinlock_profiling.cc:243] Waited 5.02 ms on lock 0x7b48000fc0d8. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52de94d7f1 00007f52de9448e7 00007f52de94477a 00007f52dea3895f 00007f52dea38627 00007f52dea4cb2c 00007f52dea4cad8 00007f52dea4ca58 00007f52dea4ca13 00007f52dea4bcbc 00007f52dd9cdbe6
0812 19:14:05.685344 (+300184us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:14:05.685401 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:06.088209 (+402808us) spinlock_profiling.cc:243] Waited 36 ms on lock 0x7b54001af754. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:06.088290 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:06.088400 (+ 110us) raft_consensus.cc:1671] finished
0812 19:14:06.088419 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:06.088729 (+ 310us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5385,"raft.run_cpu_time_us":3058,"raft.run_wall_time_us":436684,"spinlock_wait_cycles":94419072,"thread_start_us":681408,"threads_started":1,"wal-append.queue_time_us":733800}
W20260812 19:14:06.099251 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 63354.5us
I20260812 19:14:06.104662 21243 raft_consensus.cc:2379] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 5: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:14:06.104195 21328 raft_consensus.cc:1226] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:14:06.113555 21203 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:06.098345 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:06.102998 24838 consensus_queue.cc:1035] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260812 19:14:06.099673 24877 raft_consensus.cc:2781] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:14:06.135046 24877 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:14:06.136611 24877 raft_consensus.cc:3037] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:14:06.094550 24606 raft_consensus.cc:3032] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Stepping down as leader of term 7
I20260812 19:14:06.138989 24606 raft_consensus.cc:729] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:06.141275 24606 consensus_queue.cc:260] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:06.149194 24838 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:06.150861 24838 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:06.152784 25097 log.cc:924] Time spent T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.262s user 0.001s sys 0.000s
I20260812 19:14:06.160184 24825 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:14:06.195883 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 89281.9us
I20260812 19:14:06.206241 25036 raft_consensus.cc:2726] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:14:06.207597 25078 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:06.208632 25078 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:06.215767 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 55805.9us
I20260812 19:14:06.194232 24838 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:06.265321 22008 leader_election.cc:334] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.603s (SENT)
I20260812 19:14:06.266034 22008 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:06.267433 25101 raft_consensus.cc:2726] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260812 19:14:06.304827 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 77037.1us
I20260812 19:14:06.313508 25031 consensus_queue.cc:1035] T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:14:06.370996 21321 raft_consensus.cc:1226] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [term 6 LEADER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:14:06.380090 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:06.381208 22053 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
W20260812 19:14:06.391060 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 342095us
I20260812 19:14:06.404182 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:06.412555 19738 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260812 19:14:06.413226 24606 raft_consensus.cc:3037] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:06.424724 21288 leader_election.cc:302] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:14:06.402482 24877 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:06.414052 25078 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:06.447121 24990 consensus_meta.cc:287] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.544s user 0.012s sys 0.002s
I20260812 19:14:06.434356 24844 raft_consensus.cc:2781] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:14:06.425155 24674 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:06.457091 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 47127.1us
I20260812 19:14:06.453568 24844 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:06.487607 24843 consensus_queue.cc:1035] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 19:14:06.461598 24877 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:06.477074 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1773) took 1096 ms. Trace:
I20260812 19:14:06.478384 21203 leader_election.cc:398] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7.
I20260812 19:14:06.490334 24844 raft_consensus.cc:3037] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:06.506130 21151 raft_consensus.cc:1226] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 5. Current term is 6. Ops: []
I20260812 19:14:06.500164 21238 rpcz_store.cc:276] 0812 19:14:05.380751 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:05.386749 (+ 5998us) service_pool.cc:227] Handling call
0812 19:14:06.476520 (+1089771us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54000ff0d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:06.476781 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":578842496}
I20260812 19:14:06.456856 24841 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:06.476583 24991 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:06.463419 24990 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:06.539647 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:06.548947 22011 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:06.551551 21154 raft_consensus.cc:1226] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 8. Current term is 9. Ops: []
I20260812 19:14:06.554194 22053 raft_consensus.cc:1261] T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260812 19:14:06.519042 25031 consensus_queue.cc:1046] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
W20260812 19:14:06.569279 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 91214.2us
I20260812 19:14:06.570215 21149 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54796 (request call id 1708) took 1425 ms. Trace:
I20260812 19:14:06.563675 24674 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:06.571892 21149 rpcz_store.cc:276] 0812 19:14:05.103440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:05.104886 (+ 1446us) service_pool.cc:227] Handling call
0812 19:14:05.105356 (+ 470us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:05.617636 (+512280us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:06.527728 (+910092us) spinlock_profiling.cc:243] Waited 511 ms on lock 0x7b5400215054. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:06.529090 (+ 1362us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1176142080}
I20260812 19:14:06.576969 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:06.570231 24942 raft_consensus.cc:2726] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:06.574579 24971 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:14:06.585775 22008 leader_election.cc:334] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.871s (SENT)
I20260812 19:14:06.572947 24938 consensus_queue.cc:1035] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.010s
I20260812 19:14:06.590564 21204 leader_election.cc:398] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Vote denied by peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579) with higher term. Message: Invalid argument: T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7.
I20260812 19:14:06.655061 21204 leader_election.cc:401] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260812 19:14:06.589051 24971 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:06.656490 25112 log.cc:924] Time spent T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.123s user 0.000s sys 0.001s
W20260812 19:14:06.658236 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 89608us
I20260812 19:14:06.659660 21287 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:06.667232 24841 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:06.589385 24991 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:06.658223 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 74525.1us
W20260812 19:14:06.676815 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 92918.6us
I20260812 19:14:06.678155 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:06.679934 24825 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:06.680842 24825 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:14:06.655361 24821 consensus_meta.cc:287] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.061s user 0.016s sys 0.008s
W20260812 19:14:06.679952 22008 leader_election.cc:334] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.912s (SENT)
W20260812 19:14:06.685527 25111 log.cc:924] Time spent T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.175s user 0.000s sys 0.001s
I20260812 19:14:06.692658 25031 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:06.694482 25031 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:06.695317 24997 raft_consensus.cc:2781] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 19:14:06.695926 24997 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:06.696413 24997 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:06.707931 22011 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:06.710536 21287 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:14:06.710769 22008 leader_election.cc:334] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.966s (SENT)
I20260812 19:14:06.711417 22008 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:06.711923 21287 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:06.712702 25099 raft_consensus.cc:2726] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:06.713281 25029 raft_consensus.cc:2726] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579) with higher term. Message: Invalid argument: T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7.
I20260812 19:14:06.714293 24838 raft_consensus.cc:2726] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260812 19:14:06.715090 22046 raft_consensus.cc:1261] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260812 19:14:06.702594 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:06.717571 21153 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:14:06.727604 22053 raft_consensus.cc:1226] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4. Ops: []
I20260812 19:14:06.716332 24971 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:06.741613 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:06.736554 24825 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:06.742955 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:14:06.744289 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 45064.1us
I20260812 19:14:06.749023 24820 raft_consensus.cc:3032] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:14:06.749549 24820 raft_consensus.cc:729] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:06.742985 21151 raft_consensus.cc:1226] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [term 9 LEADER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 8. Current term is 9. Ops: []
I20260812 19:14:06.755479 25031 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:06.756278 25031 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:06.757012 25031 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:06.736826 24877 raft_consensus.cc:2726] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260812 19:14:06.721484 24838 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:06.759208 24838 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:06.759292 24820 consensus_queue.cc:260] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:06.760061 25095 log.cc:924] Time spent T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 1.002s user 0.012s sys 0.000s
I20260812 19:14:06.760989 24820 raft_consensus.cc:3037] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:06.761845 21324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1889) took 1134 ms. Trace:
I20260812 19:14:06.763079 21150 raft_consensus.cc:1226] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 7. Current term is 8. Ops: []
I20260812 19:14:06.763151 21324 rpcz_store.cc:276] 0812 19:14:05.627437 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:05.630445 (+ 3008us) service_pool.cc:227] Handling call
0812 19:14:05.634446 (+ 4001us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:05.634855 (+ 409us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:05.634891 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:05.635425 (+ 534us) log.cc:842] Serialized 26 byte log entry
0812 19:14:05.757080 (+121655us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:14:05.757120 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:05.757202 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:06.150446 (+393244us) spinlock_profiling.cc:243] Waited 36.2 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:14:06.759679 (+609233us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b24000c9f98. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d69e2cf4 00007f52d69e0dfb 00007f52d6a6f559 00007f52d6a6f1e5 00007f52d6a6e82d 00007f52d971146e 00007f52d9728c0c 00007f52d9727f36 00007f52d9727b58 00007f52d9731551
0812 19:14:06.761423 (+ 1744us) raft_consensus.cc:1671] finished
0812 19:14:06.761451 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:06.761636 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1904,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"spinlock_wait_cycles":90714112,"thread_start_us":121198,"threads_started":1,"wal-append.queue_time_us":121696}
I20260812 19:14:06.744114 21329 raft_consensus.cc:1261] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260812 19:14:06.766853 21329 pending_rounds.cc:76] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412: Aborting all ops after (but not including) 2
I20260812 19:14:06.767172 21329 pending_rounds.cc:97] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260812 19:14:06.767486 21329 raft_consensus.cc:2866] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260812 19:14:06.769708 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 45478.9us
I20260812 19:14:06.770514 25031 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:06.771016 22051 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 LEARNER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260812 19:14:06.772349 21287 leader_election.cc:302] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:06.758419 21153 raft_consensus.cc:1226] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 6. Current term is 7. Ops: []
I20260812 19:14:06.775089 24971 raft_consensus.cc:2781] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:14:06.775611 24971 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:06.776028 24971 raft_consensus.cc:3037] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:14:06.467381 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:14:06.761979 24838 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:06.804878 25099 consensus_queue.cc:1046] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260812 19:14:06.806841 25023 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:06.808549 24942 consensus_queue.cc:1046] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 19:14:06.809533 24877 consensus_queue.cc:1046] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260812 19:14:06.829103 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:06.811689 21237 raft_consensus.cc:1226] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 7. Current term is 8. Ops: []
I20260812 19:14:06.833884 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:06.797840 25029 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:06.801246 24990 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:06.845047 24980 raft_consensus.cc:3032] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:14:06.841099 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:06.817914 24993 raft_consensus.cc:3032] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:14:06.851773 21154 raft_consensus.cc:1226] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260812 19:14:06.852521 21319 raft_consensus.cc:1226] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [term 8 LEADER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 7. Current term is 8. Ops: []
I20260812 19:14:06.852838 21239 raft_consensus.cc:1261] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260812 19:14:06.853660 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:14:06.854040 21036 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58998 (request call id 141) took 20203 ms (client timeout 15000 ms). Trace:
I20260812 19:14:06.853679 21149 raft_consensus.cc:3037] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:14:06.855094 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:14:06.857203 25110 log.cc:924] Time spent T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.423s user 0.000s sys 0.005s
I20260812 19:14:06.858155 21327 raft_consensus.cc:1226] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
I20260812 19:14:06.840617 24821 consensus_queue.cc:1035] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260812 19:14:06.873518 21324 raft_consensus.cc:1261] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260812 19:14:06.875713 21237 raft_consensus.cc:1226] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 7. Current term is 8. Ops: []
I20260812 19:14:06.863430 21243 raft_consensus.cc:1261] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260812 19:14:06.871647 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:06.854507 21319 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:06.859030 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:06.890833 21200 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:14:06.865339 22056 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40598 (request call id 741) took 1235 ms. Trace:
I20260812 19:14:06.871868 21244 raft_consensus.cc:1261] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260812 19:14:06.915196 22056 rpcz_store.cc:276] 0812 19:14:05.629348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:05.629937 (+ 589us) service_pool.cc:227] Handling call
0812 19:14:06.864912 (+1234975us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54004e02d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:06.865117 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2411559296}
W20260812 19:14:06.853706 24832 consensus_meta.cc:287] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.648s user 0.015s sys 0.006s
I20260812 19:14:06.867455 21241 raft_consensus.cc:1226] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260812 19:14:06.871630 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:06.881014 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:06.858485 24982 consensus_queue.cc:1035] T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
I20260812 19:14:06.863014 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:06.850564 24993 raft_consensus.cc:729] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:06.929813 21329 raft_consensus.cc:2379] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 6: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:14:06.851256 25029 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:14:06.861882 25117 log.cc:924] Time spent T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.187s user 0.001s sys 0.000s
I20260812 19:14:06.932293 21244 raft_consensus.cc:1261] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260812 19:14:06.851123 24980 raft_consensus.cc:729] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:14:06.933238 24832 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:06.949723 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:06.951794 21324 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 8
W20260812 19:14:06.854528 21036 rpcz_store.cc:269] 0812 19:13:46.659258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:13:46.662437 (+ 3179us) service_pool.cc:227] Handling call
0812 19:14:06.846800 (+20184363us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0812 19:13:57.646768 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0812 19:13:57.647678 (+ 910us) write_op.cc:432] Acquiring schema lock in shared mode
0812 19:13:57.647758 (+ 80us) write_op.cc:435] Acquired schema lock
0812 19:13:57.647778 (+ 20us) tablet.cc:660] Decoding operations
0812 19:13:57.649750 (+ 1972us) write_op.cc:620] Acquiring the partition lock for write op
0812 19:13:57.649882 (+ 132us) write_op.cc:641] Partition lock acquired for write op
0812 19:13:57.649925 (+ 43us) tablet.cc:683] Acquiring locks for 14 operations
0812 19:13:57.775882 (+125957us) tablet.cc:699] Row locks acquired
0812 19:13:57.775954 (+ 72us) write_op.cc:260] PREPARE: finished
0812 19:13:57.776243 (+ 289us) write_op.cc:270] Start()
0812 19:13:57.776499 (+ 256us) write_op.cc:276] Timestamp: P: 1786562037776126 usec, L: 0
0812 19:13:57.776559 (+ 60us) op_driver.cc:331] REPLICATION: starting
0812 19:13:57.792020 (+ 15461us) log.cc:842] Serialized 5705 byte log entry
0812 19:13:58.188163 (+396143us) op_driver.cc:444] REPLICATION: finished
0812 19:13:58.671006 (+482843us) write_op.cc:301] APPLY: starting
0812 19:13:58.671129 (+ 123us) tablet.cc:1365] starting BulkCheckPresence
0812 19:13:58.671428 (+ 299us) tablet.cc:1368] finished BulkCheckPresence
0812 19:13:58.671581 (+ 153us) tablet.cc:1370] starting ApplyRowOperation cycle
0812 19:13:58.708763 (+ 37182us) tablet.cc:1381] finished ApplyRowOperation cycle
0812 19:13:58.709192 (+ 429us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0812 19:13:58.709238 (+ 46us) write_op.cc:312] APPLY: finished
0812 19:13:58.709667 (+ 429us) log.cc:842] Serialized 101 byte log entry
0812 19:13:58.710807 (+ 1140us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:13:58.711003 (+ 196us) write_op.cc:454] Released schema lock
0812 19:13:58.711088 (+ 85us) write_op.cc:341] FINISH: Updating metrics
0812 19:13:58.736415 (+ 25327us) write_op.cc:489] Releasing partition, row and schema locks
0812 19:13:58.771445 (+ 35030us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3868265,"reported_tablets":29,"child_traces":[["op",{"apply.queue_time_us":457994,"apply.run_cpu_time_us":5518,"apply.run_wall_time_us":111604,"num_ops":14,"prepare.queue_time_us":746644,"prepare.run_cpu_time_us":17843,"prepare.run_wall_time_us":237836,"raft.queue_time_us":122668,"raft.run_cpu_time_us":27635,"raft.run_wall_time_us":379299,"replication_time_us":381026,"thread_start_us":606102,"threads_started":4,"wal-append.queue_time_us":97286,"wal-append.run_cpu_time_us":20296,"wal-append.run_wall_time_us":1842034}]]}
I20260812 19:14:06.935807 21114 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:14:06.866075 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:06.870980 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:06.943928 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:06.941695 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:06.859144 25099 raft_consensus.cc:3032] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Stepping down as leader of term 5
W20260812 19:14:06.954512 22008 leader_election.cc:334] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.307s (SENT)
I20260812 19:14:06.952152 21149 raft_consensus.cc:2454] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:14:06.955214 22008 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:06.956470 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:14:06.947533 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 142230us
I20260812 19:14:06.957540 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:06.952171 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:06.959156 21149 raft_consensus.cc:1261] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260812 19:14:06.960558 21153 raft_consensus.cc:2354] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7.
I20260812 19:14:06.956259 21321 raft_consensus.cc:2454] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:14:06.963725 21318 raft_consensus.cc:2454] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:14:06.959764 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:06.967561 21329 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:06.970125 21241 raft_consensus.cc:1226] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 7. Current term is 8. Ops: []
I20260812 19:14:06.923768 24838 consensus_queue.cc:1046] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260812 19:14:06.973591 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:06.975261 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:06.976621 25029 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:06.980836 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 136283us
I20260812 19:14:06.983798 24980 consensus_queue.cc:260] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:06.954105 25099 raft_consensus.cc:729] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:14:06.849917 25023 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:06.994448 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:06.999795 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.000488 21318 raft_consensus.cc:2454] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:14:07.005411 25099 consensus_queue.cc:260] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:07.007354 25099 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:07.007426 21158 raft_consensus.cc:1226] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260812 19:14:07.008436 21112 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:07.008646 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:07.009635 24982 raft_consensus.cc:2726] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260812 19:14:07.009850 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 53674.1us
I20260812 19:14:07.011155 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:07.012827 24965 raft_consensus.cc:2781] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:14:07.013645 24965 raft_consensus.cc:686] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:07.015022 24965 consensus_queue.cc:237] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:07.017758 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:07.018426 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:07.021253 24993 consensus_queue.cc:260] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:07.022673 24993 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:07.023001 24997 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:07.027215 25124 consensus_queue.cc:1046] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260812 19:14:07.028195 24832 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:07.027575 21033 catalog_manager.cc:5495] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 3 to 5, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 5 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } }
I20260812 19:14:07.028299 25124 consensus_queue.cc:1225] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer f741102bfe1441d1b453e5a838649059 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260812 19:14:06.974020 24843 raft_consensus.cc:2726] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260812 19:14:07.044586 21322 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
I20260812 19:14:07.047456 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:07.048259 21327 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:14:07.057184 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.058768 21112 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:07.061854 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:07.054406 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.078171 21033 catalog_manager.cc:5495] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:14:07.078424 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:07.170742 21237 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:07.084039 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:07.063126 24980 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:07.075042 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:07.097445 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.173518 21327 raft_consensus.cc:2379] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 6: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:14:07.173942 21319 raft_consensus.cc:2454] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 8.
I20260812 19:14:07.045279 24997 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:07.072844 25131 raft_consensus.cc:3032] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 7 LEADER]: Stepping down as leader of term 7
I20260812 19:14:07.089491 24292 raft_consensus.cc:3032] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 19:14:07.096203 21243 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:07.108417 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.186187 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:07.185756 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.051230 24877 raft_consensus.cc:2726] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:07.184471 22046 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Advancing to term 9
I20260812 19:14:07.185973 25010 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:07.186895 21318 raft_consensus.cc:2362] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7. Re-sending same reply.
W20260812 19:14:07.175832 21204 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.70.132:39579 timed out after 2.088s (SENT) [suppressed 36 similar messages]
I20260812 19:14:07.187691 21319 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:14:07.194984 21323 raft_consensus.cc:1226] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260812 19:14:07.195514 25010 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:07.196698 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1911) took 1080 ms. Trace:
I20260812 19:14:07.199149 21329 raft_consensus.cc:2396] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260812 19:14:07.172780 24942 raft_consensus.cc:3032] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 LEADER]: Stepping down as leader of term 8
I20260812 19:14:07.209772 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:07.209671 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 319109us
I20260812 19:14:07.197153 21323 rpcz_store.cc:276] 0812 19:14:06.116525 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:06.862062 (+745537us) service_pool.cc:227] Handling call
0812 19:14:07.174662 (+312600us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:07.195557 (+ 20895us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:07.196476 (+ 919us) spinlock_profiling.cc:243] Waited 312 ms on lock 0x7b540023f0d0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:07.196633 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":717657984}
I20260812 19:14:07.210745 21329 raft_consensus.cc:2354] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 8. Current term is 9.
I20260812 19:14:07.207293 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:07.188010 24844 consensus_meta.cc:287] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.679s user 0.014s sys 0.004s
I20260812 19:14:07.208809 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:07.211375 21320 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.011s sys 0.042s
W20260812 19:14:07.219740 21320 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.515s user 0.011s sys 0.042s
I20260812 19:14:07.213210 21324 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:14:07.222549 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.224888 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.182749 25131 raft_consensus.cc:729] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:14:07.219691 21239 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 4.
I20260812 19:14:07.226588 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.250088 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.227125 21319 raft_consensus.cc:2379] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 6: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:14:07.187367 24825 raft_consensus.cc:2781] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260812 19:14:07.227443 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 196142us
I20260812 19:14:07.232913 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.187783 24292 raft_consensus.cc:729] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:07.198302 24832 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:07.250298 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.218842 24838 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:14:07.267488 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 56578.6us
I20260812 19:14:07.263556 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.209916 24942 raft_consensus.cc:729] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:14:07.272631 21150 raft_consensus.cc:1226] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 10 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 9. Current term is 10. Ops: []
I20260812 19:14:07.276232 21327 raft_consensus.cc:2396] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260812 19:14:07.281152 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:07.229372 24938 raft_consensus.cc:2781] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:07.198302 24990 consensus_queue.cc:1046] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20260812 19:14:07.344667 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:07.346927 21323 raft_consensus.cc:2454] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
I20260812 19:14:07.350147 21318 raft_consensus.cc:2379] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 8: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
W20260812 19:14:07.352205 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 82920.9us
I20260812 19:14:07.352406 21200 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:07.353057 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.353989 21324 raft_consensus.cc:2354] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 7. Current term is 8.
I20260812 19:14:07.265146 24825 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:07.348079 21329 raft_consensus.cc:2362] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7. Re-sending same reply.
I20260812 19:14:07.360101 21114 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:07.228367 24844 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:07.371768 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 417269us
W20260812 19:14:07.371110 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 200343us
I20260812 19:14:07.376438 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.278888 24991 consensus_queue.cc:1035] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 1, Time since last communication: 0.098s
I20260812 19:14:07.273049 24820 consensus_queue.cc:1046] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.092s
I20260812 19:14:07.360229 25044 consensus_queue.cc:1035] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.093s
I20260812 19:14:07.345247 24832 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:07.364558 25023 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:07.376370 21158 raft_consensus.cc:1261] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Refusing update from remote peer d1e1da0b9585460e8c365bf55679d4fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260812 19:14:07.379292 21114 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:07.384130 21328 raft_consensus.cc:1226] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 5. Current term is 6. Ops: []
I20260812 19:14:07.370151 24825 raft_consensus.cc:3037] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:07.358080 24997 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:07.398442 21321 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 9.
I20260812 19:14:07.415264 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:07.419286 22053 raft_consensus.cc:1226] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4. Ops: []
I20260812 19:14:07.429354 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:07.377990 25135 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:07.428572 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:14:07.438436 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 477470us
I20260812 19:14:07.439316 22011 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:07.378548 25138 raft_consensus.cc:2781] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260812 19:14:07.458143 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 46736.5us
I20260812 19:14:07.420702 25010 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:07.353621 24838 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
W20260812 19:14:07.482870 22008 leader_election.cc:334] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.996s (SENT)
I20260812 19:14:07.434186 25126 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:07.415863 24843 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.053s
I20260812 19:14:07.498847 22008 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:07.493512 21329 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 5.
I20260812 19:14:07.494307 21240 raft_consensus.cc:1226] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
W20260812 19:14:07.477258 25127 log.cc:924] Time spent T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.474s user 0.002s sys 0.000s
W20260812 19:14:07.486337 24971 consensus_meta.cc:287] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.706s user 0.013s sys 0.012s
I20260812 19:14:07.434791 24942 consensus_queue.cc:260] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:07.463079 25138 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:07.515323 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:07.470975 24877 raft_consensus.cc:2781] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 19:14:07.435874 24997 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:07.544365 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 43817.4us
I20260812 19:14:07.542186 21243 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:14:07.512399 21241 raft_consensus.cc:1261] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260812 19:14:07.521200 24991 raft_consensus.cc:2726] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:14:07.378459 24292 consensus_queue.cc:260] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:07.609122 24292 raft_consensus.cc:3037] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
W20260812 19:14:07.514039 25122 log.cc:924] Time spent T db5ddd3c989548e5a199029ccf42040e P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260812 19:14:07.508277 25137 consensus_queue.cc:1035] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.103s
I20260812 19:14:07.610594 25151 consensus_queue.cc:1035] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:14:07.611178 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:07.514771 25128 log.cc:924] Time spent T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.244s user 0.000s sys 0.001s
I20260812 19:14:07.612028 21318 raft_consensus.cc:2379] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 8: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
W20260812 19:14:07.516929 25136 log.cc:924] Time spent T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20260812 19:14:07.612828 22051 raft_consensus.cc:1226] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 8. Current term is 9. Ops: []
W20260812 19:14:07.518520 25129 log.cc:924] Time spent T e6eaa29a6fe94976a6124729431bd8c8 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.327s user 0.002s sys 0.000s
I20260812 19:14:07.612986 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:07.391469 25131 consensus_queue.cc:260] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:07.613905 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 2137) took 1052 ms. Trace:
I20260812 19:14:07.550060 21322 raft_consensus.cc:2396] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260812 19:14:07.614293 21320 rpcz_store.cc:276] 0812 19:14:06.561124 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:06.562971 (+ 1847us) service_pool.cc:227] Handling call
0812 19:14:06.572925 (+ 9954us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:06.588280 (+ 15355us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:06.588306 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:06.683787 (+ 95481us) log.cc:842] Serialized 26 byte log entry
0812 19:14:07.220639 (+536852us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:14:07.220681 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:07.220787 (+ 106us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:07.613688 (+392901us) raft_consensus.cc:1671] finished
0812 19:14:07.613720 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:07.613847 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13038,"raft.queue_time_us":2274,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":67779,"thread_start_us":515516,"threads_started":1,"wal-append.queue_time_us":662078}
I20260812 19:14:07.615494 25031 raft_consensus.cc:3032] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Stepping down as leader of term 7
I20260812 19:14:07.615943 25031 raft_consensus.cc:729] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:07.616372 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.617226 21320 raft_consensus.cc:2354] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for earlier term 8. Current term is 9.
I20260812 19:14:07.616904 25031 consensus_queue.cc:260] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:07.617924 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:07.618155 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:07.621994 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 108371us
I20260812 19:14:07.627547 22046 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 9.
I20260812 19:14:07.624858 22047 raft_consensus.cc:2454] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:14:07.579393 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:07.547621 24993 consensus_meta.cc:287] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.524s user 0.014s sys 0.000s
I20260812 19:14:07.592672 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:07.597020 24838 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:07.597448 24820 raft_consensus.cc:2726] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260812 19:14:07.597285 24821 consensus_queue.cc:1035] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260812 19:14:07.597586 24942 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:07.576829 25138 raft_consensus.cc:3037] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Advancing to term 10
I20260812 19:14:07.598654 21033 catalog_manager.cc:5495] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } }
I20260812 19:14:07.599625 24997 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:07.600354 21319 raft_consensus.cc:1261] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260812 19:14:07.604748 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.647495 21327 raft_consensus.cc:2379] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 6: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:14:07.515744 25004 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:07.607450 21323 raft_consensus.cc:1261] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260812 19:14:07.607905 25147 raft_consensus.cc:3032] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Stepping down as leader of term 7
I20260812 19:14:07.518597 25126 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:07.655799 21324 raft_consensus.cc:1261] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260812 19:14:07.592093 21235 raft_consensus.cc:1226] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 7. Current term is 8. Ops: []
I20260812 19:14:07.507480 25135 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:07.517937 24844 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:07.623257 25124 raft_consensus.cc:2726] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260812 19:14:07.624392 25134 log.cc:924] Time spent T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.268s user 0.001s sys 0.000s
I20260812 19:14:07.624550 21329 raft_consensus.cc:1226] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 7. Current term is 8. Ops: []
W20260812 19:14:07.622810 24996 consensus_queue.cc:1159] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "63db10063a7d4eb2a9cc9dc952654706" responder_term: 4 status { last_received { term: 3 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260812 19:14:07.619925 25148 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 19:14:07.636832 21203 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
W20260812 19:14:07.633256 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 290145us
I20260812 19:14:07.639098 22053 raft_consensus.cc:2454] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:14:07.654685 25023 raft_consensus.cc:482] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:07.646512 21319 pending_rounds.cc:76] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412: Aborting all ops after (but not including) 5
I20260812 19:14:07.676013 21319 pending_rounds.cc:97] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260812 19:14:07.676645 21319 raft_consensus.cc:2866] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:14:07.568239 24971 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:14:07.665323 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 149117us
I20260812 19:14:07.649874 25147 raft_consensus.cc:729] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:07.673941 21288 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:14:07.645279 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:07.660497 24938 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:07.631649 24877 raft_consensus.cc:686] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:14:07.677798 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:07.677232 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:07.674675 25148 consensus_queue.cc:1225] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer d1e1da0b9585460e8c365bf55679d4fa log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260812 19:14:07.700582 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:07.655381 25004 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:07.666885 24606 consensus_queue.cc:1046] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260812 19:14:07.714304 21033 catalog_manager.cc:5495] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 1 to 3, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to f741102bfe1441d1b453e5a838649059 (127.19.70.129). New cstate: current_term: 3 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } }
I20260812 19:14:07.696005 24938 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:07.722527 22055 raft_consensus.cc:1261] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260812 19:14:07.702023 24838 raft_consensus.cc:2781] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260812 19:14:07.718782 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 339587us
W20260812 19:14:07.724555 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 60241.3us
I20260812 19:14:07.728376 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:07.728608 25154 consensus_queue.cc:1046] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.007s
W20260812 19:14:07.735427 25125 log.cc:924] Time spent T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.544s user 0.001s sys 0.002s
I20260812 19:14:07.735922 25133 raft_consensus.cc:2781] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:14:07.736297 24935 raft_consensus.cc:482] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:07.736948 25031 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:07.736799 25045 consensus_queue.cc:1035] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.070s
I20260812 19:14:07.804525 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 384) took 1017 ms. Trace:
W20260812 19:14:07.738648 24980 consensus_meta.cc:287] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.558s user 0.006s sys 0.005s
I20260812 19:14:07.739650 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:07.807651 21238 rpcz_store.cc:276] 0812 19:14:06.787341 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:06.794733 (+ 7392us) service_pool.cc:227] Handling call
0812 19:14:06.804270 (+ 9537us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:06.804902 (+ 632us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:06.804942 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:06.805567 (+ 625us) log.cc:842] Serialized 26 byte log entry
0812 19:14:06.942379 (+136812us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:14:06.942427 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:06.942499 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:07.712842 (+770343us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:14:07.804118 (+ 91276us) raft_consensus.cc:1671] finished
0812 19:14:07.804157 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:07.804452 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1306,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":137,"spinlock_wait_cycles":27519744,"thread_start_us":136355,"threads_started":1,"wal-append.queue_time_us":246062}
I20260812 19:14:07.809850 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:07.811249 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:07.811762 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:07.812376 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:07.812996 21154 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:07.813362 21156 raft_consensus.cc:1226] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260812 19:14:07.739766 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:07.814829 21156 raft_consensus.cc:1226] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260812 19:14:07.815183 21152 raft_consensus.cc:1261] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260812 19:14:07.740495 24971 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:07.740644 25126 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:07.741703 22046 raft_consensus.cc:1261] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260812 19:14:07.742266 25147 consensus_queue.cc:260] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:07.817610 25147 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:07.746274 25135 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:07.748322 21114 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:07.748564 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:07.749118 25044 consensus_queue.cc:1046] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:14:07.750330 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:07.750511 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:07.751958 25131 raft_consensus.cc:3037] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:07.681386 25023 raft_consensus.cc:504] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:07.752933 24832 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:07.726774 24991 consensus_queue.cc:1046] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260812 19:14:07.727495 24838 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:07.839048 24838 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:07.840566 25023 leader_election.cc:288] T cab2f57c1cbc44b6976aeb480e73ad2c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:07.755214 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:07.756594 24877 consensus_queue.cc:237] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:07.757275 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:07.841858 22052 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:14:07.753969 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:07.767648 25158 consensus_queue.cc:1035] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260812 19:14:07.773248 21112 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:07.773721 21033 catalog_manager.cc:5495] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 4 to 5, leader changed from d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130) to 93578836968649f6bfcaecc709871412 (127.19.70.131). New cstate: current_term: 5 leader_uuid: "93578836968649f6bfcaecc709871412" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260812 19:14:07.781069 24825 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:07.781764 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:07.792835 21200 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:07.797825 25133 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:07.851534 21151 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:07.803128 24935 raft_consensus.cc:504] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:07.806819 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 170072us
W20260812 19:14:07.821938 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 73706us
I20260812 19:14:07.825127 21157 raft_consensus.cc:2379] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 8: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:14:07.832800 21237 consensus_meta.cc:287] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.659s user 0.013s sys 0.000s
I20260812 19:14:07.833791 25153 raft_consensus.cc:2726] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260812 19:14:07.836231 25099 consensus_meta.cc:287] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.828s user 0.013s sys 0.002s
I20260812 19:14:07.838127 25044 raft_consensus.cc:3032] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:14:07.840670 24991 raft_consensus.cc:3032] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Stepping down as leader of term 7
I20260812 19:14:07.768185 25004 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:07.853673 21288 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
W20260812 19:14:07.856220 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 83093.6us
I20260812 19:14:07.869491 22052 raft_consensus.cc:1261] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260812 19:14:07.856981 21237 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 9.
I20260812 19:14:07.866968 21150 raft_consensus.cc:1226] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 5. Current term is 6. Ops: []
I20260812 19:14:07.861716 21033 catalog_manager.cc:5495] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "f741102bfe1441d1b453e5a838649059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } }
I20260812 19:14:07.843967 24980 raft_consensus.cc:3032] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 LEADER]: Stepping down as leader of term 7
I20260812 19:14:07.855623 25133 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:07.842584 24832 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:07.874219 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:07.878038 21203 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:14:07.879240 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 396409us
I20260812 19:14:07.883442 22045 raft_consensus.cc:1261] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260812 19:14:07.884420 24935 leader_election.cc:288] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:07.888363 24938 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:07.861152 25044 raft_consensus.cc:729] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:07.868953 24965 raft_consensus.cc:2781] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 19:14:07.905776 25023 raft_consensus.cc:2781] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:14:07.908237 24965 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:07.908586 25023 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:07.908813 24965 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:07.909133 25023 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:07.911296 25151 consensus_queue.cc:1035] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260812 19:14:07.867275 24991 raft_consensus.cc:729] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:14:07.875063 21287 connection.cc:232] RPC call timeout handler was delayed by -1.1381229589460418s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15200000000000002s for a response
I20260812 19:14:07.905479 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:07.874063 24980 raft_consensus.cc:729] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
W20260812 19:14:08.045765 21287 connection.cc:232] RPC call timeout handler was delayed by -1.0841229589423165s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15800000000000003s for a response
I20260812 19:14:07.904548 21154 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 9.
I20260812 19:14:07.892346 21112 leader_election.cc:302] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:14:08.048369 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 166207us
W20260812 19:14:07.964593 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 48440.6us
W20260812 19:14:08.046273 21287 connection.cc:232] RPC call timeout handler was delayed by -1.0193229590076953s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1652s for a response
W20260812 19:14:08.049602 21287 connection.cc:232] RPC call timeout handler was delayed by -1.005822959006764s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16670000000000001s for a response
I20260812 19:14:08.056591 21114 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:14:08.059331 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 179710us
I20260812 19:14:07.876577 24997 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:07.879232 24971 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:07.891206 24996 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:07.879511 24820 consensus_queue.cc:1035] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.180s
I20260812 19:14:07.906780 24825 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:08.070446 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 190317us
W20260812 19:14:08.071183 22008 leader_election.cc:334] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.937s (SENT)
I20260812 19:14:08.072587 22008 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:08.064002 24990 raft_consensus.cc:2781] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:14:08.071643 21287 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:08.059494 24993 raft_consensus.cc:2726] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:14:08.090951 21112 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:14:08.092052 25169 log.cc:924] Time spent T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260812 19:14:08.091895 25158 raft_consensus.cc:2726] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 19:14:08.092911 22008 leader_election.cc:334] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.624s (SENT)
I20260812 19:14:08.095096 24825 consensus_queue.cc:1035] T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260812 19:14:08.098963 24980 consensus_queue.cc:260] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:08.100425 24938 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:07.907511 24292 consensus_queue.cc:1035] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.162s
W20260812 19:14:08.102152 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 360360us
I20260812 19:14:07.890985 24832 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:14:08.123090 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 46676.5us
I20260812 19:14:08.130355 21318 raft_consensus.cc:2454] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:14:08.138413 22008 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:08.105870 24990 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:08.017680 25044 consensus_queue.cc:260] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:08.069301 24997 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:08.071995 24996 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:14:08.149094 25170 log.cc:924] Time spent T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
W20260812 19:14:08.161134 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 368383us
I20260812 19:14:08.139190 24606 raft_consensus.cc:3032] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Stepping down as leader of term 6
W20260812 19:14:08.167022 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 103756us
I20260812 19:14:08.147369 24993 raft_consensus.cc:2726] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260812 19:14:08.083745 24971 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:08.137415 24844 raft_consensus.cc:2726] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:08.173445 22047 raft_consensus.cc:2396] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260812 19:14:08.173338 21287 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
W20260812 19:14:08.174234 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 81374.2us
W20260812 19:14:08.174691 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 315311us
I20260812 19:14:08.174887 21112 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:08.164418 25168 raft_consensus.cc:2726] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:14:08.176612 24832 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:08.177129 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:08.177624 21114 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:14:08.183141 24821 raft_consensus.cc:2726] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260812 19:14:08.183692 24997 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:08.185479 21204 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:08.186393 25167 consensus_queue.cc:1046] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260812 19:14:08.187137 24996 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:08.188089 21237 pending_rounds.cc:76] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa: Aborting all ops after (but not including) 5
I20260812 19:14:08.215682 21237 pending_rounds.cc:97] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260812 19:14:08.216116 21237 raft_consensus.cc:2866] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:14:08.171348 25004 raft_consensus.cc:2781] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election won for term 8
I20260812 19:14:08.196552 22050 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
W20260812 19:14:08.198292 22008 leader_election.cc:334] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.836s (SENT)
I20260812 19:14:08.200508 21203 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:08.210399 25044 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:08.172176 24606 raft_consensus.cc:729] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:08.210901 25135 consensus_queue.cc:1046] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:14:08.212090 25010 consensus_queue.cc:1046] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260812 19:14:08.161504 24990 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:14:08.224447 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 53148.2us
I20260812 19:14:08.150013 24991 consensus_queue.cc:260] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:08.233783 24292 raft_consensus.cc:2726] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:14:08.227455 24993 raft_consensus.cc:2781] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader election won for term 9
W20260812 19:14:08.256345 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 83104.9us
I20260812 19:14:08.225482 25147 raft_consensus.cc:2726] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260812 19:14:08.241627 25138 consensus_meta.cc:287] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.578s user 0.009s sys 0.003s
I20260812 19:14:08.269920 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:08.217370 25029 consensus_queue.cc:1046] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260812 19:14:08.272369 21319 raft_consensus.cc:1226] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 5. Current term is 6. Ops: []
I20260812 19:14:08.270841 21322 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
W20260812 19:14:08.333475 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 133037us
I20260812 19:14:08.339387 21203 leader_election.cc:398] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T c6309c277c104b23a58d4824d764d397 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7.
I20260812 19:14:08.241281 24832 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:08.351645 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:08.357391 21318 raft_consensus.cc:1226] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 7. Current term is 8. Ops: []
W20260812 19:14:08.360457 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 80488.4us
W20260812 19:14:08.361347 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 163130us
I20260812 19:14:08.252775 24982 consensus_queue.cc:1046] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.093s
I20260812 19:14:08.372351 25150 raft_consensus.cc:3032] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Stepping down as leader of term 7
I20260812 19:14:08.372839 25150 raft_consensus.cc:729] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:08.373726 25150 consensus_queue.cc:260] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:08.376443 21153 raft_consensus.cc:1261] T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260812 19:14:08.240767 25023 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:08.341831 24991 raft_consensus.cc:3037] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:08.416592 21287 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:14:08.347666 25029 consensus_queue.cc:1225] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer f741102bfe1441d1b453e5a838649059 log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260812 19:14:08.345675 24993 raft_consensus.cc:686] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [term 9 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:14:08.426667 21151 consensus_meta.cc:287] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.569s user 0.012s sys 0.000s
I20260812 19:14:08.432885 21151 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
W20260812 19:14:08.473819 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 57338.9us
I20260812 19:14:08.414825 25183 raft_consensus.cc:2781] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260812 19:14:08.477149 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 257850us
I20260812 19:14:08.283063 25138 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:08.353089 24832 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:08.280056 24606 consensus_queue.cc:260] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:14:08.468338 24965 consensus_meta.cc:287] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.526s user 0.013s sys 0.001s
I20260812 19:14:08.425892 24843 raft_consensus.cc:2726] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260812 19:14:08.508639 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:08.472090 25153 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:08.467773 25004 raft_consensus.cc:686] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [term 8 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:08.420444 25161 consensus_queue.cc:1035] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.066s
I20260812 19:14:08.470413 24996 raft_consensus.cc:2781] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260812 19:14:08.518944 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 238523us
I20260812 19:14:08.517508 21324 raft_consensus.cc:2454] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 8.
I20260812 19:14:08.488693 25183 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:08.530181 24606 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:08.530875 24971 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:08.532229 21329 raft_consensus.cc:1226] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
I20260812 19:14:08.534551 21287 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:08.534915 24993 consensus_queue.cc:237] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:08.525079 25153 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:08.536082 21287 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:08.537987 25153 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:08.510005 24292 raft_consensus.cc:482] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:08.548884 21112 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:14:08.544152 24838 consensus_meta.cc:287] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.705s user 0.014s sys 0.000s
I20260812 19:14:08.562474 21329 raft_consensus.cc:1226] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
I20260812 19:14:08.527038 24996 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:08.528388 25150 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
W20260812 19:14:08.564464 22008 leader_election.cc:334] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.542s (SENT)
I20260812 19:14:08.541340 25183 raft_consensus.cc:3037] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:08.575282 24820 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:08.576368 25138 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 10 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:08.577119 25178 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:08.577723 24832 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:08.552748 24292 raft_consensus.cc:504] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:08.579216 25023 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:08.580504 21112 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:14:08.581157 25147 raft_consensus.cc:2781] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260812 19:14:08.569550 25010 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:08.520905 24965 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:08.569820 24838 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:08.571085 24996 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Advancing to term 9
W20260812 19:14:08.588438 25181 log.cc:924] Time spent T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.101s user 0.001s sys 0.000s
W20260812 19:14:08.574011 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 233068us
I20260812 19:14:08.590571 21287 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
W20260812 19:14:08.610278 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 115165us
I20260812 19:14:08.613389 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:08.557900 24997 raft_consensus.cc:2726] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260812 19:14:08.626773 22008 leader_election.cc:334] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.385s (SENT)
I20260812 19:14:08.634807 22055 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:08.634944 22008 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:08.634878 21204 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:14:08.643397 25189 log.cc:924] Time spent T 5405717c8c644382be14b4f6cd0a3bd7 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260812 19:14:08.643966 24292 leader_election.cc:288] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:14:08.644876 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 64524.2us
I20260812 19:14:08.629081 24980 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:08.616735 25147 raft_consensus.cc:482] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:08.650512 25147 raft_consensus.cc:3037] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Advancing to term 10
I20260812 19:14:08.651548 25201 raft_consensus.cc:2781] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:14:08.652081 25201 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:14:08.652503 25201 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:08.639814 25023 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:08.655112 25023 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:08.656217 24965 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:08.636845 24990 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:08.657433 25023 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:08.659466 25167 raft_consensus.cc:3032] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 8 LEADER]: Stepping down as leader of term 8
I20260812 19:14:08.659982 25167 raft_consensus.cc:729] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:08.661020 25167 consensus_queue.cc:260] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:08.608253 24971 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:08.532279 25004 consensus_queue.cc:237] T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:08.662408 25167 raft_consensus.cc:3037] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Advancing to term 9
W20260812 19:14:08.693656 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 66951.3us
W20260812 19:14:08.697992 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 130181us
I20260812 19:14:08.640812 25131 raft_consensus.cc:2726] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:08.705582 21203 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:08.619122 24820 raft_consensus.cc:504] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:08.699321 25202 raft_consensus.cc:2781] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:14:08.710994 21287 leader_election.cc:302] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:14:08.693265 24938 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:08.739981 25202 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:08.636399 25010 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:08.627584 25178 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:08.707532 25199 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:14:08.803890 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 66731.5us
I20260812 19:14:08.688704 24971 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:14:08.843562 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 138146us
I20260812 19:14:08.865240 21203 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:14:08.867184 22008 leader_election.cc:334] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.876s (SENT)
I20260812 19:14:08.867722 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:08.868651 25193 raft_consensus.cc:2726] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260812 19:14:08.869867 22008 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:08.796662 25138 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:08.736289 25151 consensus_queue.cc:1046] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.092s
I20260812 19:14:08.879793 21243 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
W20260812 19:14:08.873225 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 164584us
I20260812 19:14:08.805069 24838 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:08.810786 25202 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:08.876906 24832 raft_consensus.cc:2781] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:08.842177 25045 raft_consensus.cc:2726] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260812 19:14:08.895265 25133 consensus_meta.cc:287] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.980s user 0.014s sys 0.005s
I20260812 19:14:08.924466 21114 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:08.871733 25031 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:08.886730 25168 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:14:08.946758 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 79602.6us
I20260812 19:14:08.951747 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:14:08.980548 22008 leader_election.cc:334] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.050s (SENT)
I20260812 19:14:08.981817 22008 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:08.901162 24292 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:08.827478 24938 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:08.803651 25198 consensus_queue.cc:1046] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.186s
I20260812 19:14:08.995651 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:08.996307 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:14:08.996577 25205 log.cc:924] Time spent T 00000000000000000000000000000000 P 61e1609653694f899be42137777c67c1: Append to log took a long time: real 0.333s user 0.001s sys 0.001s
I20260812 19:14:08.944801 24838 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:08.944806 24832 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:08.946064 24990 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:09.037659 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 168318us
I20260812 19:14:08.938902 25138 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:09.093158 19738 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260812 19:14:08.844102 25199 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:14:09.112478 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 401565us
I20260812 19:14:08.956496 24606 consensus_queue.cc:1046] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.135s
I20260812 19:14:08.996582 25218 raft_consensus.cc:2726] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:14:09.089859 24832 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:09.130537 21243 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:08.957815 25133 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.088581 24844 raft_consensus.cc:2726] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:14:08.980249 25031 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:09.152195 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:09.012398 24292 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.167333 21112 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:09.178279 21240 raft_consensus.cc:1226] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260812 19:14:09.090602 24838 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.165112 25151 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:09.000253 25168 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.191660 25225 raft_consensus.cc:2781] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:14:09.195979 25225 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:09.196382 25225 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:09.196723 25235 raft_consensus.cc:2726] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:14:09.197492 25235 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:09.168838 24820 leader_election.cc:288] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:09.203761 22055 consensus_meta.cc:287] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.562s user 0.008s sys 0.002s
I20260812 19:14:09.212675 22055 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:14:09.210232 21243 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:14:09.198035 25235 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.188874 25218 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:09.256920 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:14:09.234741 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 67535.1us
I20260812 19:14:09.233057 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:09.258509 21112 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:14:09.224520 25204 log.cc:924] Time spent T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.394s user 0.001s sys 0.000s
I20260812 19:14:09.268429 21112 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:09.218889 25135 raft_consensus.cc:3032] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 9 LEADER]: Stepping down as leader of term 9
I20260812 19:14:09.270885 25135 raft_consensus.cc:729] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:09.271628 21238 raft_consensus.cc:858] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }:
New config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:09.272177 21238 raft_consensus.cc:1565] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Could not prepare op 'term: 5 index: 5' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }:
New config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.271937 25135 consensus_queue.cc:260] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.273756 25135 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Advancing to term 10
I20260812 19:14:09.274935 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:09.273679 25133 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:09.285702 21112 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:09.287249 25194 raft_consensus.cc:3032] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Stepping down as leader of term 7
I20260812 19:14:09.291770 24292 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:09.293800 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 393) took 1109 ms. Trace:
I20260812 19:14:09.216460 24971 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:09.233043 25237 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:09.306985 21287 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:09.252694 25010 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:09.297156 25212 log.cc:924] Time spent T 8c13f33db56241a7938cfe52798e77a2 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20260812 19:14:09.312371 21288 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:14:09.306681 21240 rpcz_store.cc:276] 0812 19:14:08.184145 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:08.184340 (+ 195us) service_pool.cc:227] Handling call
0812 19:14:08.184925 (+ 585us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:09.178818 (+993893us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:09.293479 (+114661us) spinlock_profiling.cc:243] Waited 993 ms on lock 0x7b5400041ad4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:09.293688 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2283760000}
W20260812 19:14:09.298336 24991 consensus_meta.cc:287] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.871s user 0.010s sys 0.007s
I20260812 19:14:09.211575 25151 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:09.297657 24990 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
W20260812 19:14:09.342501 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 57114.7us
I20260812 19:14:09.346511 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:09.343338 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:14:09.252856 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 382062us
I20260812 19:14:09.303783 25194 raft_consensus.cc:729] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:09.347438 21240 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:14:09.352900 25176 raft_consensus.cc:2726] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260812 19:14:09.349184 24991 raft_consensus.cc:2726] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260812 19:14:09.311347 25044 raft_consensus.cc:2781] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:14:09.362630 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:09.363968 21112 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:14:09.347923 22050 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
W20260812 19:14:09.377935 22008 leader_election.cc:334] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 2.128s (SENT)
I20260812 19:14:09.371842 25215 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:09.331084 24938 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:09.374022 22053 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 677) took 1207 ms. Trace:
I20260812 19:14:09.378485 22008 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:09.376247 22055 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:14:09.384793 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:09.385857 21288 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:14:09.396899 21288 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:14:09.362027 24292 raft_consensus.cc:2781] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:14:09.270265 25218 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.395442 21238 raft_consensus.cc:2932] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } } }
I20260812 19:14:09.373513 25138 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:09.390007 22053 rpcz_store.cc:276] 0812 19:14:08.155098 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:08.155270 (+ 172us) service_pool.cc:227] Handling call
0812 19:14:08.168683 (+ 13413us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:08.190750 (+ 22067us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:08.190798 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:08.200683 (+ 9885us) log.cc:842] Serialized 26 byte log entry
0812 19:14:08.564058 (+363375us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:14:08.564128 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:08.564240 (+ 112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:09.362316 (+798076us) raft_consensus.cc:1671] finished
0812 19:14:09.362341 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:09.362481 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2243,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":117,"thread_start_us":348765,"threads_started":1,"wal-append.queue_time_us":470518}
I20260812 19:14:09.374796 25044 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:14:09.404222 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 120242us
W20260812 19:14:09.389716 24996 consensus_meta.cc:287] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.750s user 0.014s sys 0.000s
I20260812 19:14:09.411307 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:09.357252 24990 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:09.420991 21237 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.679s user 0.000s sys 0.014s
W20260812 19:14:09.425979 21237 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.684s user 0.000s sys 0.014s
I20260812 19:14:09.428077 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:09.428972 21321 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
I20260812 19:14:09.428340 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:09.430315 21329 raft_consensus.cc:3037] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:09.399307 24971 raft_consensus.cc:2781] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:09.403975 24938 raft_consensus.cc:2781] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260812 19:14:09.421373 25004 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.707s user 0.012s sys 0.004s
I20260812 19:14:09.409143 25044 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:09.347904 25237 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:09.408551 24292 raft_consensus.cc:686] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [term 6 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
W20260812 19:14:09.450052 24980 consensus_meta.cc:287] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.803s user 0.013s sys 0.001s
I20260812 19:14:09.463390 25151 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:09.449653 25004 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s user 0.012s sys 0.004s
I20260812 19:14:09.454340 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:09.412689 25031 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:09.475268 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:09.476320 21323 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:09.484740 21329 raft_consensus.cc:2454] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:14:09.442418 24971 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:09.488493 21233 raft_consensus.cc:2454] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8.
I20260812 19:14:09.489018 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:09.413926 24996 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:09.394068 25215 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.449613 24825 raft_consensus.cc:3032] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:14:09.424161 24820 consensus_queue.cc:1035] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20260812 19:14:09.466895 24838 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:09.531455 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 118117us
I20260812 19:14:09.466383 24832 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:14:09.483508 25147 consensus_meta.cc:287] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.832s user 0.011s sys 0.002s
I20260812 19:14:09.494127 25176 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:09.488641 25157 raft_consensus.cc:2781] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:14:09.496167 24292 consensus_queue.cc:237] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.444585 24938 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:09.506877 21240 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:14:09.508328 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:09.512806 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 134911us
I20260812 19:14:09.517868 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:09.556885 21233 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 6
I20260812 19:14:09.487931 25031 raft_consensus.cc:482] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:09.524662 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:09.522054 25219 raft_consensus.cc:2726] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:14:09.540762 21327 raft_consensus.cc:1226] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 8. Current term is 9. Ops: []
W20260812 19:14:09.521780 25211 log.cc:924] Time spent T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.319s user 0.000s sys 0.002s
I20260812 19:14:09.404974 25194 consensus_queue.cc:260] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:09.562098 21324 raft_consensus.cc:2379] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 7: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
W20260812 19:14:09.540441 25183 consensus_meta.cc:287] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.923s user 0.014s sys 0.000s
W20260812 19:14:09.540117 25210 log.cc:924] Time spent T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.180s user 0.001s sys 0.000s
W20260812 19:14:09.596633 25207 log.cc:924] Time spent T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.370s user 0.001s sys 0.001s
I20260812 19:14:09.501796 24971 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:09.598955 25246 consensus_queue.cc:1035] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.175s
I20260812 19:14:09.599855 25246 consensus_queue.cc:1225] T c31c6a2accba4d29ba096500b9d1a911 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer 93578836968649f6bfcaecc709871412 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260812 19:14:09.601897 25252 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:09.602972 21318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 2158) took 1185 ms. Trace:
I20260812 19:14:09.603525 25237 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:09.603389 21318 rpcz_store.cc:276] 0812 19:14:08.417368 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:08.418323 (+ 955us) service_pool.cc:227] Handling call
0812 19:14:08.419306 (+ 983us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:08.427157 (+ 7851us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:08.427205 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:08.479035 (+ 51830us) log.cc:842] Serialized 26 byte log entry
0812 19:14:08.628792 (+149757us) raft_consensus.cc:1631] Marking committed up to 2
0812 19:14:08.628831 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:08.629954 (+ 1123us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:09.602705 (+972751us) raft_consensus.cc:1671] finished
0812 19:14:09.602728 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:09.602901 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9594,"raft.queue_time_us":11782,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":36144,"thread_start_us":103447,"threads_started":1,"wal-append.queue_time_us":667520}
W20260812 19:14:09.606359 25251 log.cc:924] Time spent T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260812 19:14:09.552543 25176 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:09.607440 25182 log.cc:924] Time spent T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 1.112s user 0.007s sys 0.000s
I20260812 19:14:09.609155 22045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40624 (request call id 676) took 1521 ms. Trace:
I20260812 19:14:09.610217 25176 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:09.547397 25157 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:09.611181 25157 raft_consensus.cc:3037] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:09.611291 25168 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:09.612061 25168 consensus_queue.cc:1035] T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 19:14:09.613945 25235 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:09.563306 21320 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:09.561491 24938 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:09.596181 25149 raft_consensus.cc:2781] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader election won for term 8
I20260812 19:14:09.602556 25252 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.520481 24825 raft_consensus.cc:729] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:09.547945 25147 raft_consensus.cc:504] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:09.586566 25232 raft_consensus.cc:2726] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260812 19:14:09.658716 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:09.532122 25249 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:09.567408 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:14:09.662910 25167 consensus_meta.cc:287] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.968s user 0.011s sys 0.004s
W20260812 19:14:09.705302 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 157825us
I20260812 19:14:09.609539 22045 rpcz_store.cc:276] 0812 19:14:08.087690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:08.094080 (+ 6390us) service_pool.cc:227] Handling call
0812 19:14:08.094481 (+ 401us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:14:08.094980 (+ 499us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:08.095011 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:14:08.095696 (+ 685us) log.cc:842] Serialized 52 byte log entry
0812 19:14:08.225307 (+129611us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:14:08.225359 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:08.225435 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:09.516883 (+1291448us) spinlock_profiling.cc:243] Waited 7.82 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:14:09.608715 (+ 91832us) raft_consensus.cc:1671] finished
0812 19:14:09.608740 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:09.609080 (+ 340us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1489,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":118,"spinlock_wait_cycles":17976704,"thread_start_us":124344,"threads_started":1,"wal-append.queue_time_us":253351}
I20260812 19:14:09.706176 21112 leader_election.cc:302] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:09.700129 21324 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:09.676496 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:09.707754 21318 raft_consensus.cc:2454] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4.
I20260812 19:14:09.677224 21233 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:14:09.708837 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:09.675522 25194 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:09.584903 25031 raft_consensus.cc:504] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.713145 25176 raft_consensus.cc:2781] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:14:09.653616 25199 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:09.653384 25183 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.660851 25218 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:09.682260 25254 log.cc:924] Time spent T 0569d19f4a06475c84bdc61b1c5f549f P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
W20260812 19:14:09.683313 25250 log.cc:924] Time spent T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20260812 19:14:09.674995 25178 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:09.685971 25215 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:09.701413 21320 raft_consensus.cc:1226] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:14:09.700593 21240 raft_consensus.cc:2379] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 8: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:09.565842 21328 raft_consensus.cc:1226] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 7. Current term is 8. Ops: []
I20260812 19:14:09.710098 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:09.737265 21233 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8.
I20260812 19:14:09.738435 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:09.739902 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:09.741107 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:14:09.708065 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 207533us
I20260812 19:14:09.711500 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:09.743072 21321 raft_consensus.cc:2362] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 4. Re-sending same reply.
I20260812 19:14:09.744081 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:09.745666 21321 raft_consensus.cc:1261] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Refusing update from remote peer 63db10063a7d4eb2a9cc9dc952654706: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260812 19:14:09.715131 21239 raft_consensus.cc:1261] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260812 19:14:09.719307 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:09.719942 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:09.721879 24996 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:09.722910 25147 leader_election.cc:288] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 10 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:09.747727 21318 raft_consensus.cc:2396] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260812 19:14:09.724834 25249 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.750469 25249 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:09.727296 25183 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:09.719373 24990 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:09.730896 21235 raft_consensus.cc:1226] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 7. Current term is 8. Ops: []
W20260812 19:14:09.733999 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 54710.2us
I20260812 19:14:09.754190 21287 leader_election.cc:302] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:09.742296 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:09.723970 25252 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:14:09.783227 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 66201.4us
I20260812 19:14:09.783711 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:14:09.776636 25150 consensus_meta.cc:287] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.195s user 0.012s sys 0.004s
I20260812 19:14:09.787041 21112 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:09.772279 24996 raft_consensus.cc:2726] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:14:09.776129 25260 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:09.807017 25260 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:09.807456 25225 consensus_meta.cc:287] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.611s user 0.011s sys 0.000s
I20260812 19:14:09.808621 25031 leader_election.cc:288] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:09.810644 25260 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:09.795252 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:09.793241 25147 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:09.813653 25147 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.815373 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:09.816118 25147 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:09.801065 24843 raft_consensus.cc:3032] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:14:09.774106 25176 raft_consensus.cc:686] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:09.818359 25176 consensus_queue.cc:237] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:09.819427 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1809) took 1084 ms. Trace:
I20260812 19:14:09.820585 24971 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.776285 25149 raft_consensus.cc:686] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [term 8 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:09.821530 22045 raft_consensus.cc:1261] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260812 19:14:09.822191 25149 consensus_queue.cc:237] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.807535 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:09.828049 24832 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:09.828296 25023 consensus_queue.cc:1035] T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260812 19:14:09.808305 25225 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:09.847004 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:09.847800 22055 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Advancing to term 8
W20260812 19:14:09.820017 25201 consensus_meta.cc:287] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.167s user 0.015s sys 0.000s
I20260812 19:14:09.817385 24843 raft_consensus.cc:729] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:09.778383 25215 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:09.816130 21150 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
I20260812 19:14:09.805691 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:14:09.788372 25135 consensus_meta.cc:287] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.512s user 0.011s sys 0.002s
I20260812 19:14:09.819854 21239 rpcz_store.cc:276] 0812 19:14:08.735207 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:08.735449 (+ 242us) service_pool.cc:227] Handling call
0812 19:14:08.735899 (+ 450us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:09.746796 (+1010897us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:09.819110 (+ 72314us) spinlock_profiling.cc:243] Waited 979 ms on lock 0x7b5400150254. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:09.819356 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2251115776}
I20260812 19:14:09.858949 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:09.859730 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:09.860368 21239 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:09.861366 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:09.862063 22052 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
W20260812 19:14:09.863305 25266 log.cc:924] Time spent T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260812 19:14:09.863484 25150 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:09.864112 25150 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:09.866921 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:09.867134 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:09.866371 25150 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:09.842801 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 55870us
I20260812 19:14:09.868764 25178 raft_consensus.cc:482] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:09.843396 25151 raft_consensus.cc:2781] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 19:14:09.869236 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:09.869704 25151 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:09.869369 25178 raft_consensus.cc:504] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:09.870050 21153 raft_consensus.cc:3037] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:14:09.870155 25151 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:09.843438 24971 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:09.780499 24838 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:09.871145 24838 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:09.871804 25178 leader_election.cc:288] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:09.871917 25044 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:09.873427 24838 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:09.848515 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:09.873980 25044 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:09.874804 25225 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:09.874536 25201 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.876259 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:09.848690 21323 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
W20260812 19:14:09.813097 25228 log.cc:924] Time spent T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.637s user 0.002s sys 0.000s
W20260812 19:14:09.853624 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 99645.3us
I20260812 19:14:09.854511 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:09.855444 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:09.857023 25215 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.867136 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:09.869132 21112 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:09.842047 25260 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:09.782224 24825 consensus_queue.cc:260] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:09.877122 21154 raft_consensus.cc:2379] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 8: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:10.090821 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:09.973886 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:10.060266 21153 raft_consensus.cc:2454] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:14:10.092342 21156 raft_consensus.cc:2379] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 10: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:14:09.987027 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 44956.8us
I20260812 19:14:09.974215 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:09.912108 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:09.912992 22011 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:09.973876 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:14:10.046985 21238 consensus_meta.cc:287] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.641s user 0.011s sys 0.002s
I20260812 19:14:09.942371 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:09.942673 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:10.091629 21150 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 7.
I20260812 19:14:10.105877 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:10.107537 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:10.110313 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:09.883899 22055 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8.
I20260812 19:14:10.109045 21321 raft_consensus.cc:1226] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 5. Current term is 6. Ops: []
I20260812 19:14:10.113667 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:10.113576 24844 consensus_queue.cc:1035] T db5ddd3c989548e5a199029ccf42040e P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260812 19:14:10.114352 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:10.104650 21155 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:14:10.093705 22051 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:09.942322 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:09.931190 21115 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:10.109162 21200 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:10.115820 21149 raft_consensus.cc:2379] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 6: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
W20260812 19:14:10.112210 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 243138us
I20260812 19:14:10.111295 21158 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8.
W20260812 19:14:10.124671 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 193623us
W20260812 19:14:10.074541 25227 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.814s user 0.006s sys 0.009s
W20260812 19:14:10.079406 25202 consensus_meta.cc:287] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.091s user 0.012s sys 0.001s
I20260812 19:14:10.125725 21287 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:10.140203 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:10.126579 21112 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:10.128412 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:10.134910 21239 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:14:09.974510 24292 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:14:10.131811 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 218953us
I20260812 19:14:10.139146 21115 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:10.010771 25264 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:10.143507 25259 consensus_queue.cc:1046] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.013s
I20260812 19:14:10.144279 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1816) took 1336 ms. Trace:
W20260812 19:14:10.135044 25227 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.877s user 0.007s sys 0.009s
I20260812 19:14:10.145593 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:10.095114 24838 raft_consensus.cc:482] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:10.180128 22011 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:10.140568 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:10.149760 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:10.151345 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:10.153272 21318 raft_consensus.cc:1226] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:14:10.186533 21321 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 8.
I20260812 19:14:10.154819 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:10.144878 25270 raft_consensus.cc:2781] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 19:14:10.188743 24844 raft_consensus.cc:2726] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260812 19:14:10.188838 25270 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:10.189445 21156 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:14:10.189746 25270 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:10.163750 21288 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:14:10.137864 25225 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260812 19:14:10.191075 21329 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.004s sys 0.012s
I20260812 19:14:10.169970 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:10.172113 21233 raft_consensus.cc:3037] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:10.140182 25161 raft_consensus.cc:2781] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:10.175849 21240 rpcz_store.cc:276] 0812 19:14:08.807524 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:10.143262 (+1335738us) service_pool.cc:227] Handling call
0812 19:14:10.143677 (+ 415us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:10.143880 (+ 203us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:10.143904 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:10.143943 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:14:10.143980 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:10.144067 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:10.144203 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:10.144822 25201 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:10.183450 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 76654.3us
W20260812 19:14:10.191506 21329 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.502s user 0.004s sys 0.012s
I20260812 19:14:10.173763 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1817) took 1365 ms. Trace:
I20260812 19:14:09.929193 25157 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:10.196722 21320 raft_consensus.cc:2379] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 7: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:14:10.197670 22052 raft_consensus.cc:2354] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 7. Current term is 8.
W20260812 19:14:10.193169 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 66735.9us
I20260812 19:14:10.105521 25260 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:10.202430 21287 leader_election.cc:302] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:10.102152 21243 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:14:10.394933 21243 raft_consensus.cc:595] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:14:10.267997 21115 leader_election.cc:398] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Vote denied by peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579) with higher term. Message: Invalid argument: T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 7. Current term is 8.
I20260812 19:14:10.396201 21115 leader_election.cc:401] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260812 19:14:10.205302 21239 rpcz_store.cc:276] 0812 19:14:08.807912 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:10.172583 (+1364671us) service_pool.cc:227] Handling call
0812 19:14:10.173079 (+ 496us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:10.173324 (+ 245us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:10.173361 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:10.173417 (+ 56us) raft_consensus.cc:1631] Marking committed up to 3
0812 19:14:10.173467 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:10.173534 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:10.173685 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:10.397279 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:10.397930 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:10.192651 25264 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:10.398605 21156 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:14:10.399454 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:10.400682 21153 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8.
I20260812 19:14:10.177984 24292 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:10.295696 22053 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
W20260812 19:14:10.303526 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 101235us
I20260812 19:14:10.195935 24838 raft_consensus.cc:504] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:10.171201 25202 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:10.295022 25260 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:10.304191 24832 raft_consensus.cc:2726] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 19:14:10.203006 25161 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:10.179468 25149 raft_consensus.cc:2781] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:10.195101 25225 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:10.273070 21324 raft_consensus.cc:1226] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260812 19:14:10.302785 25215 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:10.309949 22052 raft_consensus.cc:1261] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260812 19:14:10.461092 21241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1814) took 1658 ms. Trace:
W20260812 19:14:10.319324 25213 log.cc:924] Time spent T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 1.323s user 0.005s sys 0.000s
I20260812 19:14:10.199120 24844 raft_consensus.cc:2781] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader election won for term 8
I20260812 19:14:10.416443 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5ddd3c989548e5a199029ccf42040e" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:10.291975 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:10.324407 25157 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:10.314443 24938 consensus_meta.cc:287] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.639s user 0.003s sys 0.006s
I20260812 19:14:10.475610 24938 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:10.478173 22052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:10.240204 25227 raft_consensus.cc:2781] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:14:10.200026 25265 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 19:14:10.480214 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
W20260812 19:14:10.403855 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 1.05133e+06us
W20260812 19:14:10.395578 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 1.08365e+06us
I20260812 19:14:10.508266 22050 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Advancing to term 9
I20260812 19:14:10.403023 21287 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:10.393620 21236 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1815) took 1591 ms. Trace:
I20260812 19:14:10.405745 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:10.412570 25237 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:10.400318 25264 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:10.418931 21233 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 9.
W20260812 19:14:10.480003 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 370932us
I20260812 19:14:10.517989 21233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1659) took 1163 ms. Trace:
I20260812 19:14:10.386513 25031 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:10.269196 21329 pending_rounds.cc:76] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412: Aborting all ops after (but not including) 4
I20260812 19:14:10.521178 21329 pending_rounds.cc:97] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260812 19:14:10.521483 21329 raft_consensus.cc:2866] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260812 19:14:10.389053 25208 log.cc:924] Time spent T cab2f57c1cbc44b6976aeb480e73ad2c P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 1.091s user 0.000s sys 0.003s
I20260812 19:14:10.466048 21320 raft_consensus.cc:2454] T 85aa3ef35cc045a393fcbb13f15370a7 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
W20260812 19:14:10.391177 25164 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.325s user 0.023s sys 0.025s
I20260812 19:14:10.472544 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:10.461582 21241 rpcz_store.cc:276] 0812 19:14:08.802278 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:08.820292 (+ 18014us) service_pool.cc:227] Handling call
0812 19:14:08.871761 (+ 51469us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:08.871992 (+ 231us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:14:09.255376 (+383384us) log.cc:842] Serialized 14 byte log entry
0812 19:14:10.460632 (+1205256us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:10.460718 (+ 86us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:14:10.460780 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:10.460865 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:10.461019 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1022444,"prepare.queue_time_us":432300,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":11088,"thread_start_us":516808,"threads_started":2,"wal-append.queue_time_us":284295}
I20260812 19:14:10.501756 21322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 2155) took 2158 ms. Trace:
I20260812 19:14:10.413203 21243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34764 (request call id 32) took 1034 ms. Trace:
I20260812 19:14:10.510185 21321 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
W20260812 19:14:10.493798 25276 log.cc:924] Time spent T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
W20260812 19:14:10.480962 25221 log.cc:924] Time spent T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 1.484s user 0.007s sys 0.003s
I20260812 19:14:10.478330 24938 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:10.501475 21235 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37894 (request call id 1643) took 1753 ms. Trace:
I20260812 19:14:10.514068 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:10.489322 24838 leader_election.cc:288] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:10.519002 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:10.481832 21244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1812) took 1680 ms. Trace:
I20260812 19:14:10.518425 21233 rpcz_store.cc:276] 0812 19:14:09.354845 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:09.856648 (+501803us) service_pool.cc:227] Handling call
0812 19:14:10.517919 (+661271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:10.554191 21327 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:10.473515 25161 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Advancing to term 8
W20260812 19:14:10.532439 25164 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.594s user 0.030s sys 0.035s
I20260812 19:14:10.520521 25215 consensus_queue.cc:1046] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20260812 19:14:10.559070 25215 raft_consensus.cc:3032] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:14:10.559643 25215 raft_consensus.cc:729] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:14:10.560693 25215 consensus_queue.cc:260] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:10.533125 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:10.564340 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:10.565076 21241 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 8.
I20260812 19:14:10.566125 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:10.533460 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:10.530324 21322 rpcz_store.cc:276] 0812 19:14:08.342962 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:08.343296 (+ 334us) service_pool.cc:227] Handling call
0812 19:14:08.353728 (+ 10432us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:08.543349 (+189621us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:08.543393 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:08.558650 (+ 15257us) log.cc:842] Serialized 26 byte log entry
0812 19:14:08.736873 (+178223us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:14:08.736916 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:09.943217 (+1206301us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b540016e9d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:09.943290 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:10.278271 (+334981us) spinlock_profiling.cc:243] Waited 4.8 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:14:10.501411 (+223140us) raft_consensus.cc:1671] finished
0812 19:14:10.501442 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:10.501644 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26788,"raft.queue_time_us":11251,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":7304,"spinlock_wait_cycles":407142144,"thread_start_us":89146,"threads_started":1,"wal-append.queue_time_us":390802}
I20260812 19:14:10.527726 25138 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:10.509572 21236 rpcz_store.cc:276] 0812 19:14:08.806927 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:08.807206 (+ 279us) service_pool.cc:227] Handling call
0812 19:14:08.865587 (+ 58381us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:08.865898 (+ 311us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:14:09.081195 (+215297us) log.cc:842] Serialized 14 byte log entry
0812 19:14:10.393058 (+1311863us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b18003ec560. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b014ca 00007f52e01d6f99 00007f52d69c5990 00007f52d6b87864 00007f52d6b87155 00007f52d9a9e3d0 00007f52d9ab0576 00007f52d9ac10d7 00007f52d9ac1058
0812 19:14:10.393240 (+ 182us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:10.393294 (+ 54us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:14:10.393349 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:10.393424 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:10.393552 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":118417,"prepare.queue_time_us":350784,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":44,"raft.queue_time_us":1131328,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":3088,"spinlock_wait_cycles":3646336,"thread_start_us":391788,"threads_started":2,"wal-append.queue_time_us":273736,"wal-append.run_cpu_time_us":1428,"wal-append.run_wall_time_us":1019013}
I20260812 19:14:10.535737 24292 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:10.416060 25198 consensus_queue.cc:1035] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.105s
I20260812 19:14:10.528393 24996 raft_consensus.cc:2726] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:14:10.530473 21243 rpcz_store.cc:276] 0812 19:14:09.371889 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:09.912526 (+540637us) service_pool.cc:227] Handling call
0812 19:14:10.406420 (+493894us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:10.473670 25149 raft_consensus.cc:482] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:14:10.514268 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 111341us
I20260812 19:14:10.480404 25284 raft_consensus.cc:2781] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:14:10.502853 24844 raft_consensus.cc:686] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [term 8 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:10.507030 25265 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:10.544468 21235 rpcz_store.cc:276] 0812 19:14:08.747917 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:08.793706 (+ 45789us) service_pool.cc:227] Handling call
0812 19:14:08.864845 (+ 71139us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:09.753427 (+888582us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:10.501081 (+747654us) spinlock_profiling.cc:243] Waited 866 ms on lock 0x7b54002571d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:10.501382 (+ 301us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1991081344}
I20260812 19:14:10.478403 25045 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:10.388381 25151 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:10.492945 25202 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:10.555217 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e242bf1e9fb45a197a8eaa86bf09f44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:10.600761 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:10.544229 25193 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:10.483242 24971 consensus_queue.cc:1035] T ae9be4809de24d32bdd888c6f9511d0f P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.075s
I20260812 19:14:10.553887 25270 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:10.569175 21241 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 8.
I20260812 19:14:10.585632 25149 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:10.534828 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:10.619059 21320 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 7
I20260812 19:14:10.518925 25227 raft_consensus.cc:686] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:10.584045 21287 leader_election.cc:302] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:10.620785 21287 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:10.621997 21287 leader_election.cc:302] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:10.473307 25225 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:10.554088 21244 rpcz_store.cc:276] 0812 19:14:08.801517 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:08.801706 (+ 189us) service_pool.cc:227] Handling call
0812 19:14:08.865228 (+ 63522us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:14:08.867131 (+ 1903us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:08.867162 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:14:08.870362 (+ 3200us) log.cc:842] Serialized 52 byte log entry
0812 19:14:09.006000 (+135638us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:14:09.448765 (+442765us) log.cc:842] Serialized 14 byte log entry
0812 19:14:09.812351 (+363586us) spinlock_profiling.cc:243] Waited 4.03 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:14:10.481233 (+668882us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:10.481345 (+ 112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:10.481407 (+ 62us) raft_consensus.cc:1671] finished
0812 19:14:10.481539 (+ 132us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:10.481746 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1036653,"prepare.queue_time_us":349546,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":1034756,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":3294,"spinlock_wait_cycles":9261952,"thread_start_us":487651,"threads_started":2,"wal-append.queue_time_us":281701}
I20260812 19:14:10.586726 25284 raft_consensus.cc:482] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:10.603578 21236 raft_consensus.cc:2454] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 9.
I20260812 19:14:10.602475 21200 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:10.608235 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:10.562286 25215 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:10.586848 25164 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:10.644752 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:10.624111 21287 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:10.541738 25133 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:10.629173 21150 raft_consensus.cc:2354] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 8. Current term is 9.
I20260812 19:14:10.629300 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:10.617795 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:10.634804 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:10.597045 25265 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:10.640262 22050 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 9.
I20260812 19:14:10.544896 25031 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:14:10.647027 21287 leader_election.cc:302] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:10.648087 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:10.649766 21114 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:10.614135 25167 consensus_queue.cc:1046] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
I20260812 19:14:10.598502 25138 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:10.612473 25045 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:10.615260 25193 raft_consensus.cc:504] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.008669 21320 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:11.135325 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbcfe527fe4701863d72b9bb480ccf" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:11.136731 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:11.137055 25198 raft_consensus.cc:482] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:11.137409 21241 raft_consensus.cc:2454] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:11.066217 21327 raft_consensus.cc:2454] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:10.635697 24292 raft_consensus.cc:2781] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:14:11.137738 25198 raft_consensus.cc:504] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.138784 25270 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:10.887259 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 478163us
I20260812 19:14:10.965776 25227 consensus_queue.cc:237] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.140933 25045 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:11.141247 25198 leader_election.cc:288] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:10.649829 21233 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 10.
I20260812 19:14:10.649169 25202 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:11.142349 25202 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:10.985146 24606 raft_consensus.cc:3032] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Stepping down as leader of term 7
I20260812 19:14:11.144117 24606 raft_consensus.cc:729] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:11.144747 21321 raft_consensus.cc:1226] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 8. Current term is 9. Ops: []
I20260812 19:14:10.982540 25176 raft_consensus.cc:3032] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 8 LEADER]: Stepping down as leader of term 8
I20260812 19:14:11.145563 25176 raft_consensus.cc:729] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:10.968164 25260 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:11.146472 25260 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:10.630188 24832 raft_consensus.cc:2781] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader election won for term 9
I20260812 19:14:11.142875 21322 raft_consensus.cc:1261] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260812 19:14:11.142654 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:11.146795 25176 consensus_queue.cc:260] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.182902 25176 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:11.183082 21157 raft_consensus.cc:2379] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 10: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:11.185938 25031 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:11.175899 21244 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:14:10.655452 25199 consensus_queue.cc:1046] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.528s
I20260812 19:14:11.148164 21288 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:14:10.729151 25244 raft_consensus.cc:2781] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 19:14:10.855937 24844 consensus_queue.cc:237] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:11.170514 25202 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:11.149087 25260 leader_election.cc:288] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:11.175710 21233 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:11.178656 25193 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:10.744994 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 477093us
I20260812 19:14:10.634356 24996 raft_consensus.cc:2781] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 19:14:11.180431 21328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1922) took 1756 ms. Trace:
I20260812 19:14:10.726058 25164 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.239456 22047 raft_consensus.cc:1261] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260812 19:14:11.239962 22047 pending_rounds.cc:76] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: Aborting all ops after (but not including) 5
I20260812 19:14:11.240237 22047 pending_rounds.cc:97] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260812 19:14:11.240049 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0569d19f4a06475c84bdc61b1c5f549f" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:11.240463 25164 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:10.654990 25191 raft_consensus.cc:482] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:11.180905 25265 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:11.241376 25191 raft_consensus.cc:504] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:11.242519 24825 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:11.242937 25265 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 9 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:11.243832 25191 leader_election.cc:288] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:10.676354 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 60046.4us
I20260812 19:14:11.224598 25244 raft_consensus.cc:482] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:11.247257 25244 raft_consensus.cc:3037] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:11.248431 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:11.249171 21233 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:14:11.250169 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:11.255306 21233 raft_consensus.cc:2379] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 8: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
W20260812 19:14:10.675028 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 72651.1us
I20260812 19:14:11.187628 25198 raft_consensus.cc:3032] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:14:10.647907 24838 raft_consensus.cc:2726] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:14:10.635933 25284 raft_consensus.cc:3037] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:11.167297 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:11.270556 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:11.271703 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:11.193969 24971 raft_consensus.cc:2781] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:10.883376 25237 raft_consensus.cc:2726] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579) with higher term. Message: Invalid argument: T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 7. Current term is 8.
I20260812 19:14:11.194489 24832 raft_consensus.cc:686] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [term 9 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:11.195930 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:11.285771 21236 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:14:11.286334 21150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1752) took 1357 ms. Trace:
W20260812 19:14:10.716112 21323 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.074s sys 0.118s
I20260812 19:14:11.197474 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:14:11.287119 21323 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.128s user 0.081s sys 0.129s
I20260812 19:14:11.287637 21322 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 9
W20260812 19:14:11.197391 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 547677us
W20260812 19:14:11.288404 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 62031.6us
I20260812 19:14:11.289515 21114 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:11.198984 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1810) took 2398 ms. Trace:
I20260812 19:14:10.871206 25184 consensus_queue.cc:1046] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.655s
I20260812 19:14:11.199647 21239 raft_consensus.cc:1203] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-6.5] Dedup: 5.4->[6.5-6.5]
I20260812 19:14:11.292613 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:11.293260 21239 raft_consensus.cc:1226] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 6. Current term is 7. Ops: [1.2-6.5]
I20260812 19:14:11.293458 21236 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Advancing to term 9
W20260812 19:14:11.200682 21237 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.777s user 0.015s sys 0.045s
W20260812 19:14:11.294196 21237 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.871s user 0.015s sys 0.045s
I20260812 19:14:11.201946 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:11.295339 21241 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:11.215466 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:11.216761 22053 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40624 (request call id 692) took 1363 ms. Trace:
I20260812 19:14:10.634296 25044 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:11.290289 21238 rpcz_store.cc:276] 0812 19:14:08.800706 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:08.800895 (+ 189us) service_pool.cc:227] Handling call
0812 19:14:08.871806 (+ 70911us) raft_consensus.cc:1457] Updating replica for 5 ops
0812 19:14:09.207876 (+336070us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:09.207952 (+ 76us) raft_consensus.cc:1525] Triggering prepare for 5 ops
0812 19:14:09.273579 (+ 65627us) log.cc:842] Serialized 558 byte log entry
0812 19:14:09.388970 (+115391us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:14:10.103800 (+714830us) log.cc:842] Serialized 14 byte log entry
0812 19:14:11.197635 (+1093835us) log.cc:842] Serialized 14 byte log entry
0812 19:14:11.198191 (+ 556us) log.cc:842] Serialized 14 byte log entry
0812 19:14:11.198464 (+ 273us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:11.198590 (+ 126us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:11.198677 (+ 87us) raft_consensus.cc:1671] finished
0812 19:14:11.198688 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:11.198879 (+ 191us) inbound_call.cc:173] Queueing success response
0812 19:14:11.247781 (+ 48902us) spinlock_profiling.cc:243] Waited 26.5 ms on lock 0x7b18002d44e0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d6b8a638 00007f52d6b8db11 00007f52d6b8dac9 00007f52d6b8da59 00007f52d6b8da21 00007f52d6b8cd1d 00007f52dfde2204 00007f52dfde2039 00007f52d6b6cdc6
Metrics: {"mutex_wait_us":403225,"prepare.queue_time_us":785450,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":496629,"raft.run_cpu_time_us":334,"raft.run_wall_time_us":12910,"spinlock_wait_cycles":151563264,"thread_start_us":764121,"threads_started":3,"wal-append.queue_time_us":223647,"wal-append.run_cpu_time_us":4431,"wal-append.run_wall_time_us":1701462}
I20260812 19:14:11.312168 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:10.679306 25133 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:11.313694 21112 leader_election.cc:302] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:14:11.234277 21235 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:14:11.319414 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:11.147230 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:11.240510 22047 raft_consensus.cc:2866] T 4c656db261bd45f8b421339ba53dd87f P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:14:11.251020 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:11.122066 25151 leader_election.cc:288] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:11.230978 21328 rpcz_store.cc:276] 0812 19:14:09.425611 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:09.438707 (+ 13096us) service_pool.cc:227] Handling call
0812 19:14:09.449418 (+ 10711us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:09.775140 (+325722us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:11.180126 (+1404986us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b540016dad4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:11.180355 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":258238208}
I20260812 19:14:11.322911 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:11.256584 22050 raft_consensus.cc:1261] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260812 19:14:11.218345 24993 raft_consensus.cc:2726] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260812 19:14:11.193240 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 546260us
I20260812 19:14:11.331594 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:11.331946 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefbee13b3b74848be07bfcab278a3c8" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:11.332443 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:11.332670 21287 leader_election.cc:302] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:11.266770 25260 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:14:11.334059 21322 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 9.
I20260812 19:14:11.334125 21287 leader_election.cc:302] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:11.223104 24292 raft_consensus.cc:686] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [term 7 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:11.287663 22055 raft_consensus.cc:2379] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 9: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:14:11.292336 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:11.296058 21327 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 9
I20260812 19:14:11.299271 25135 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:11.296073 22053 rpcz_store.cc:276] 0812 19:14:09.853608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:09.854134 (+ 526us) service_pool.cc:227] Handling call
0812 19:14:11.216549 (+1362415us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b540058e654. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:11.216688 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1002508160}
W20260812 19:14:11.361718 21287 connection.cc:232] RPC call timeout handler was delayed by -1.7294031269848347s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22340000000000002s for a response
I20260812 19:14:11.286728 22011 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:11.286864 21150 rpcz_store.cc:276] 0812 19:14:09.928910 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:10.115853 (+186943us) service_pool.cc:227] Handling call
0812 19:14:11.279815 (+1163962us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b5400181354. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:11.286260 (+ 6445us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1113837952}
I20260812 19:14:11.309300 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:10.633982 24938 raft_consensus.cc:482] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:11.258771 25283 raft_consensus.cc:2781] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader election won for term 9
I20260812 19:14:11.260972 24997 raft_consensus.cc:2726] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260812 19:14:11.322639 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37894 (request call id 1644) took 2511 ms. Trace:
I20260812 19:14:11.364837 21158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:11.187026 21153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1753) took 1257 ms. Trace:
I20260812 19:14:11.325413 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:11.365747 21158 raft_consensus.cc:2379] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 6: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:11.365634 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:11.365222 21239 rpcz_store.cc:276] 0812 19:14:08.810972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:10.397740 (+1586768us) service_pool.cc:227] Handling call
0812 19:14:11.199472 (+801732us) raft_consensus.cc:1457] Updating replica for 4 ops
0812 19:14:11.293685 (+ 94213us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:11.322385 (+ 28700us) spinlock_profiling.cc:243] Waited 801 ms on lock 0x7b5400256cd0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa5a09 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:11.322550 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1842199424}
I20260812 19:14:11.189805 24843 consensus_queue.cc:260] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:11.259841 24996 raft_consensus.cc:686] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:11.282996 24971 raft_consensus.cc:482] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:11.380997 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:11.352980 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:11.296752 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:11.333428 25138 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:11.362969 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 76404.1us
I20260812 19:14:11.305112 21200 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:11.363317 21327 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 9.
W20260812 19:14:11.363766 21287 connection.cc:232] RPC call timeout handler was delayed by -1.7186031270539388s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22460000000000002s for a response
I20260812 19:14:11.258908 25198 raft_consensus.cc:729] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:11.270982 25164 consensus_queue.cc:1046] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20260812 19:14:11.367379 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:11.424373 21239 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:14:11.323704 21233 raft_consensus.cc:2454] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:11.421144 22011 leader_election.cc:302] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:14:11.429358 25198 consensus_queue.cc:260] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:11.434450 21150 raft_consensus.cc:1226] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260812 19:14:11.368319 21112 leader_election.cc:302] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:11.325451 25237 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:11.361214 21114 leader_election.cc:302] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:14:11.444511 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 83373.5us
I20260812 19:14:11.366546 21238 raft_consensus.cc:2454] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:14:11.368153 21153 rpcz_store.cc:276] 0812 19:14:09.929269 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:10.115230 (+185961us) service_pool.cc:227] Handling call
0812 19:14:11.186789 (+1071559us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b54000c3354. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:11.186914 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":500882304}
I20260812 19:14:11.322698 25174 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:11.380308 21149 raft_consensus.cc:1226] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
W20260812 19:14:11.357250 21240 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s user 0.015s sys 0.047s
W20260812 19:14:11.452010 21240 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.834s user 0.016s sys 0.047s
W20260812 19:14:11.414582 21287 connection.cc:232] RPC call timeout handler was delayed by -1.7168031269684434s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22480000000000003s for a response
W20260812 19:14:11.359575 25221 log.cc:924] Time spent T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
I20260812 19:14:11.412356 21155 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:14:11.347168 24838 raft_consensus.cc:2781] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 19:14:11.347354 25023 raft_consensus.cc:2726] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260812 19:14:11.437314 21154 raft_consensus.cc:1226] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 10 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 9. Current term is 10. Ops: []
I20260812 19:14:11.425951 24843 raft_consensus.cc:3037] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:11.468048 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1818) took 2659 ms. Trace:
I20260812 19:14:11.445735 21114 leader_election.cc:302] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:14:11.337381 25238 consensus_queue.cc:1035] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.082s
I20260812 19:14:11.471829 21240 rpcz_store.cc:276] 0812 19:14:08.808257 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:10.196223 (+1387966us) service_pool.cc:227] Handling call
0812 19:14:10.282661 (+ 86438us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:10.282982 (+ 321us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:14:10.617870 (+334888us) log.cc:842] Serialized 14 byte log entry
0812 19:14:11.467508 (+849638us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:11.467586 (+ 78us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:14:11.467655 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:11.467750 (+ 95us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:11.467946 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14777,"prepare.queue_time_us":909083,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":18566,"thread_start_us":1168749,"threads_started":2}
I20260812 19:14:11.424564 25283 raft_consensus.cc:686] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [term 9 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:11.469799 21238 raft_consensus.cc:1226] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Rejecting Update request from peer 93578836968649f6bfcaecc709871412 for earlier term 4. Current term is 5. Ops: []
I20260812 19:14:11.452075 21235 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656"
dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:34764
I20260812 19:14:11.476083 21235 raft_consensus.cc:595] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:14:11.476603 21235 raft_consensus.cc:603] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260812 19:14:11.324852 25044 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.415175 25265 raft_consensus.cc:2781] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:14:11.414527 25268 consensus_queue.cc:1046] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20260812 19:14:11.483559 21114 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:11.413138 24971 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:11.421526 25193 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:11.462723 24838 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:11.472591 21152 raft_consensus.cc:1226] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260812 19:14:11.441155 21239 raft_consensus.cc:1226] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 10 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 9. Current term is 10. Ops: []
I20260812 19:14:11.350932 25260 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:11.479758 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37894 (request call id 1645) took 2642 ms. Trace:
I20260812 19:14:11.450999 25259 raft_consensus.cc:2781] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 19:14:11.382185 24938 raft_consensus.cc:504] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.482164 25198 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:11.374073 25291 raft_consensus.cc:2726] T db5ddd3c989548e5a199029ccf42040e P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260812 19:14:11.439745 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 71513.4us
I20260812 19:14:11.466809 25236 raft_consensus.cc:2781] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 19:14:11.386323 25135 raft_consensus.cc:504] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: NON_VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: true } }
I20260812 19:14:11.503608 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1820) took 2685 ms. Trace:
I20260812 19:14:11.504364 21238 rpcz_store.cc:276] 0812 19:14:08.837684 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.446949 (+2609265us) service_pool.cc:227] Handling call
0812 19:14:11.466974 (+ 20025us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:11.479328 (+ 12354us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:11.479646 (+ 318us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:11.472747 24965 raft_consensus.cc:2726] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260812 19:14:11.506362 21114 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706
I20260812 19:14:11.444288 25200 consensus_queue.cc:1046] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20260812 19:14:11.480370 25238 consensus_queue.cc:1225] T 45e03c303e40432e8f5da639f7edff82 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer d1e1da0b9585460e8c365bf55679d4fa log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260812 19:14:11.461339 21233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1822) took 2619 ms. Trace:
W20260812 19:14:11.520314 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 215260us
I20260812 19:14:11.530303 21233 rpcz_store.cc:276] 0812 19:14:08.819266 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.437781 (+2618515us) service_pool.cc:227] Handling call
0812 19:14:11.438273 (+ 492us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:11.438561 (+ 288us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:11.438602 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:11.438697 (+ 95us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:14:11.438756 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:11.438835 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:11.438987 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:11.532441 21233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1831) took 1002 ms. Trace:
I20260812 19:14:11.532579 21114 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:11.532866 21233 rpcz_store.cc:276] 0812 19:14:10.530049 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.531532 (+1001483us) service_pool.cc:227] Handling call
0812 19:14:11.531887 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:11.532108 (+ 221us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:11.532137 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:11.532175 (+ 38us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:14:11.532205 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:11.532271 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:11.532377 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:11.533779 21112 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:11.534734 21237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 392) took 3351 ms. Trace:
I20260812 19:14:11.462361 25174 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.512735 21320 raft_consensus.cc:3037] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:11.508049 25259 raft_consensus.cc:482] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:11.540197 21112 leader_election.cc:302] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters: d1e1da0b9585460e8c365bf55679d4fa
W20260812 19:14:11.490829 25149 consensus_meta.cc:287] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.842s user 0.014s sys 0.000s
I20260812 19:14:11.486157 25151 raft_consensus.cc:2781] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260812 19:14:11.524353 25225 consensus_meta.cc:287] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.869s user 0.011s sys 0.003s
I20260812 19:14:11.500437 25031 raft_consensus.cc:2781] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:11.505367 24838 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:11.519606 21154 raft_consensus.cc:1226] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
I20260812 19:14:11.491833 24996 consensus_queue.cc:237] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.556039 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 2160) took 3057 ms. Trace:
I20260812 19:14:11.535322 21237 rpcz_store.cc:276] 0812 19:14:08.183756 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:08.187292 (+ 3536us) service_pool.cc:227] Handling call
0812 19:14:08.187762 (+ 470us) raft_consensus.cc:1457] Updating replica for 4 ops
0812 19:14:08.432016 (+244254us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:08.432051 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 4 ops
0812 19:14:08.468215 (+ 36164us) log.cc:842] Serialized 104 byte log entry
0812 19:14:08.696128 (+227913us) raft_consensus.cc:1631] Marking committed up to 8
0812 19:14:09.426459 (+730331us) log.cc:842] Serialized 14 byte log entry
0812 19:14:10.391676 (+965217us) spinlock_profiling.cc:243] Waited 4.69 ms on lock 0x7b4800190158. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52de94d7f1 00007f52de9448e7 00007f52de94477a 00007f52dea3895f 00007f52dea38627 00007f52dea4cb2c 00007f52dea4cad8 00007f52dea4ca58 00007f52dea4ca13 00007f52dea4bcbc 00007f52dd9cdbe6
0812 19:14:11.294655 (+902979us) log.cc:842] Serialized 14 byte log entry
0812 19:14:11.516251 (+221596us) log.cc:842] Serialized 14 byte log entry
0812 19:14:11.516545 (+ 294us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:11.534263 (+ 17718us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:11.534348 (+ 85us) raft_consensus.cc:1671] finished
0812 19:14:11.534369 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:11.534597 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1124434,"prepare.queue_time_us":1274181,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":78,"raft.queue_time_us":1366857,"raft.run_cpu_time_us":6862,"raft.run_wall_time_us":1865781,"spinlock_wait_cycles":11362304,"thread_start_us":2123409,"threads_started":5,"wal-append.queue_time_us":372881,"wal-append.run_cpu_time_us":5017,"wal-append.run_wall_time_us":1727343}
W20260812 19:14:11.518354 25215 consensus_meta.cc:287] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.869s user 0.013s sys 0.000s
I20260812 19:14:11.516690 25010 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:11.483870 25265 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:11.546029 21200 leader_election.cc:302] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:14:11.520982 21239 rpcz_store.cc:276] 0812 19:14:08.818265 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.440406 (+2622141us) service_pool.cc:227] Handling call
0812 19:14:11.440876 (+ 470us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:11.503222 (+ 62346us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:11.503496 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:11.520478 25236 raft_consensus.cc:482] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:11.556509 21319 rpcz_store.cc:276] 0812 19:14:08.507522 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:08.514029 (+ 6507us) service_pool.cc:227] Handling call
0812 19:14:08.528701 (+ 14672us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:08.540583 (+ 11882us) raft_consensus.cc:1520] Early marking committed up to index 6
0812 19:14:08.693671 (+153088us) log.cc:842] Serialized 14 byte log entry
0812 19:14:10.480133 (+1786462us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x7b44000232cc. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d69cef11 00007f52d6a9ca95 00007f52d6a9d4b2 00007f52d9739511 00007f52d9734e1a 00007f52d9728bee 00007f52d9727f36 00007f52d9727b58 00007f52d9731551 00007f52d9731509 00007f52d9731499
0812 19:14:11.156321 (+676188us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b1800403780. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d973582c 00007f52d6b6c9dc 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b89e22 00007f52d97275bb 00007f52d972d063 00007f52d972d57b
0812 19:14:11.221405 (+ 65084us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:11.257576 (+ 36171us) log.cc:842] Serialized 564 byte log entry
0812 19:14:11.257834 (+ 258us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:14:11.257889 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:11.258012 (+ 123us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:11.555818 (+297806us) raft_consensus.cc:1671] finished
0812 19:14:11.555842 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:11.555980 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54510,"prepare.queue_time_us":339981,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":55002,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":12051,"spinlock_wait_cycles":58772608,"thread_start_us":328274,"threads_started":2,"wal-append.queue_time_us":290135}
I20260812 19:14:11.554035 21238 raft_consensus.cc:1261] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260812 19:14:11.535996 21233 raft_consensus.cc:1261] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260812 19:14:11.561782 21320 raft_consensus.cc:2396] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260812 19:14:11.564338 21239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 396) took 1023 ms. Trace:
I20260812 19:14:11.564787 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1826) took 1036 ms. Trace:
I20260812 19:14:11.564946 21233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1842) took 1025 ms. Trace:
I20260812 19:14:11.565443 21233 rpcz_store.cc:276] 0812 19:14:10.539918 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.533820 (+993902us) service_pool.cc:227] Handling call
0812 19:14:11.534172 (+ 352us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:11.564615 (+ 30443us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:11.564866 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:11.565215 21238 rpcz_store.cc:276] 0812 19:14:10.528545 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.528030 (+999485us) service_pool.cc:227] Handling call
0812 19:14:11.553713 (+ 25683us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:11.564519 (+ 10806us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:11.564733 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:11.567639 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:11.568562 21152 raft_consensus.cc:2379] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 9: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:11.564816 21239 rpcz_store.cc:276] 0812 19:14:10.540502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.563434 (+1022932us) service_pool.cc:227] Handling call
0812 19:14:11.563772 (+ 338us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:11.563957 (+ 185us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:11.563989 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:11.564031 (+ 42us) raft_consensus.cc:1631] Marking committed up to 8
0812 19:14:11.564070 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:11.564126 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:11.564265 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:11.569317 25010 raft_consensus.cc:504] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:11.508361 25193 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:11.547359 25246 raft_consensus.cc:2781] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:14:11.546721 24993 raft_consensus.cc:2781] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:11.551945 25259 raft_consensus.cc:3037] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:11.474716 24832 consensus_queue.cc:237] T 65db14a4838d40218947e8bd3420e892 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:11.582419 21237 raft_consensus.cc:1226] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 5. Current term is 6. Ops: []
I20260812 19:14:11.551452 25202 consensus_queue.cc:1046] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
W20260812 19:14:11.584679 25294 log.cc:924] Time spent T ad64c74a7de6449c8152bc0e093a59db P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.152s user 0.001s sys 0.000s
I20260812 19:14:11.560259 25225 raft_consensus.cc:504] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:11.586953 25285 log.cc:924] Time spent T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.235s user 0.001s sys 0.001s
I20260812 19:14:11.581261 25246 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:11.622222 25246 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:11.556200 25149 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:11.555835 25031 raft_consensus.cc:482] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:11.623267 25031 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:11.624608 25149 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:11.626183 25260 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:11.501884 24292 consensus_queue.cc:237] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.551863 25151 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:11.629611 25151 raft_consensus.cc:3037] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:11.630211 25260 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:11.631176 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:11.631022 25260 raft_consensus.cc:504] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:11.633867 25260 leader_election.cc:288] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:11.579530 25249 raft_consensus.cc:2781] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 19:14:11.634572 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:11.635018 25249 raft_consensus.cc:686] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:11.635637 21155 raft_consensus.cc:2379] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 9: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:11.636140 25249 consensus_queue.cc:237] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.636669 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:11.552795 25293 raft_consensus.cc:2781] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election won for term 7
W20260812 19:14:11.594161 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 446090us
I20260812 19:14:11.595367 21237 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1843) took 1055 ms. Trace:
I20260812 19:14:11.638773 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:11.639034 25293 raft_consensus.cc:686] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:11.639079 21237 rpcz_store.cc:276] 0812 19:14:10.540215 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.556921 (+1016706us) service_pool.cc:227] Handling call
0812 19:14:11.582230 (+ 25309us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:11.595086 (+ 12856us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:11.595298 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:11.593492 21238 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57682 (request call id 397) took 1033 ms. Trace:
I20260812 19:14:11.640075 25293 consensus_queue.cc:237] T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.640323 21238 rpcz_store.cc:276] 0812 19:14:10.559961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.566917 (+1006956us) service_pool.cc:227] Handling call
0812 19:14:11.592944 (+ 26027us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:11.593157 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:11.593182 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:11.593222 (+ 40us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:14:11.593256 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:11.593305 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:11.593424 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:11.598470 21235 raft_consensus.cc:1261] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260812 19:14:11.573256 25297 log.cc:924] Time spent T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.225s user 0.002s sys 0.000s
I20260812 19:14:11.567929 25265 raft_consensus.cc:3037] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:11.647037 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1941) took 1429 ms. Trace:
I20260812 19:14:11.647737 21329 rpcz_store.cc:276] 0812 19:14:10.217593 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:10.267968 (+ 50375us) service_pool.cc:227] Handling call
0812 19:14:10.268679 (+ 711us) raft_consensus.cc:1457] Updating replica for 2 ops
0812 19:14:10.529193 (+260514us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:10.529240 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0812 19:14:10.531599 (+ 2359us) log.cc:842] Serialized 590 byte log entry
0812 19:14:10.913195 (+381596us) raft_consensus.cc:1631] Marking committed up to 4
0812 19:14:10.913255 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:10.913364 (+ 109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:11.646716 (+733352us) raft_consensus.cc:1671] finished
0812 19:14:11.646739 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:11.646942 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":99,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":151,"thread_start_us":317130,"threads_started":1,"wal-append.queue_time_us":749649}
I20260812 19:14:11.652357 25284 raft_consensus.cc:504] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.600633 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:11.578722 25215 raft_consensus.cc:2781] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260812 19:14:11.586392 25193 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:11.580420 24993 raft_consensus.cc:482] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
W20260812 19:14:11.609608 25161 consensus_meta.cc:287] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 1.024s user 0.013s sys 0.000s
I20260812 19:14:11.581717 25303 raft_consensus.cc:2726] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260812 19:14:11.581029 25283 consensus_queue.cc:237] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:11.576686 25133 leader_election.cc:288] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:11.614003 25259 raft_consensus.cc:504] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.730273 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:11.615929 25023 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:11.616839 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:11.618361 21203 leader_election.cc:302] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:11.619364 24938 leader_election.cc:288] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:11.620987 24971 raft_consensus.cc:504] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.661238 25265 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:11.661486 21241 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:11.604611 25256 consensus_queue.cc:1046] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260812 19:14:11.663746 25244 raft_consensus.cc:504] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.669119 25225 leader_election.cc:288] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:11.670044 21151 raft_consensus.cc:3037] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:11.575166 25236 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Advancing to term 9
W20260812 19:14:11.680356 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 227263us
W20260812 19:14:11.683477 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 100889us
W20260812 19:14:11.684682 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 118330us
I20260812 19:14:11.689899 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cbc2eec68f4a328fa4cf6c0de63656" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:11.763916 21329 raft_consensus.cc:1226] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 5. Current term is 6. Ops: []
I20260812 19:14:11.667748 25151 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:11.704955 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 122686us
I20260812 19:14:11.722889 24993 raft_consensus.cc:3037] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:11.771193 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:11.772524 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
W20260812 19:14:11.723110 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 274574us
I20260812 19:14:11.727020 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:11.773895 21154 raft_consensus.cc:2379] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 9: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:11.773734 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:11.774755 21329 raft_consensus.cc:2379] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 9: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:11.775173 21319 raft_consensus.cc:1226] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 7. Current term is 8. Ops: []
I20260812 19:14:11.752451 25138 raft_consensus.cc:2726] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:14:11.688086 25202 raft_consensus.cc:3032] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Stepping down as leader of term 4
I20260812 19:14:11.729744 25176 consensus_queue.cc:1035] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260812 19:14:11.730057 25149 consensus_queue.cc:1035] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.040s
I20260812 19:14:11.736341 21152 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692"
dest_uuid: "f741102bfe1441d1b453e5a838649059"
mode: GRACEFUL
new_leader_uuid: "93578836968649f6bfcaecc709871412"
from {username='slave'} at 127.0.0.1:47804
I20260812 19:14:11.796386 21152 raft_consensus.cc:595] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 10 FOLLOWER]: Received request to transfer leadership to TS 93578836968649f6bfcaecc709871412
I20260812 19:14:11.796792 21152 raft_consensus.cc:603] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260812 19:14:11.737543 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 316564us
I20260812 19:14:11.717730 21236 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 9.
I20260812 19:14:11.602779 25238 consensus_queue.cc:1035] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.122s
W20260812 19:14:11.742908 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 124619us
I20260812 19:14:11.748272 21241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1840) took 1209 ms. Trace:
I20260812 19:14:11.709138 25215 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:11.755314 21150 raft_consensus.cc:1226] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 5. Current term is 6. Ops: []
I20260812 19:14:11.774996 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c13f33db56241a7938cfe52798e77a2" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:11.748174 25199 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:11.780771 21240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37902 (request call id 1825) took 2591 ms. Trace:
I20260812 19:14:11.782003 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:11.773826 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:11.733947 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:11.734018 21153 raft_consensus.cc:2354] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 7. Current term is 8.
I20260812 19:14:11.783342 22047 raft_consensus.cc:2354] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 8. Current term is 9.
I20260812 19:14:11.766943 25174 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:11.731935 25161 raft_consensus.cc:504] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:11.814766 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:11.823259 21244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37902 (request call id 1827) took 1294 ms. Trace:
I20260812 19:14:11.859941 21149 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:14:11.799566 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:11.859228 21241 rpcz_store.cc:276] 0812 19:14:10.539294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.199364 (+660070us) service_pool.cc:227] Handling call
0812 19:14:11.748183 (+548819us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:11.860109 21244 rpcz_store.cc:276] 0812 19:14:10.528901 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:10.626214 (+ 97313us) service_pool.cc:227] Handling call
0812 19:14:11.822997 (+1196783us) spinlock_profiling.cc:243] Waited 29 ms on lock 0x7b54002c38d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:11.823173 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":66626816}
I20260812 19:14:11.860975 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:11.861569 21327 raft_consensus.cc:2379] T 414940a0231d4db9a5cba87d50c5f122 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 9: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:11.861907 21149 raft_consensus.cc:3037] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:11.862425 21240 rpcz_store.cc:276] 0812 19:14:09.189198 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.483263 (+2294065us) service_pool.cc:227] Handling call
0812 19:14:11.503763 (+ 20500us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:11.504009 (+ 246us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:11.504036 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:11.511526 (+ 7490us) log.cc:842] Serialized 26 byte log entry
0812 19:14:11.686140 (+174614us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:14:11.686184 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:11.686269 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:11.780559 (+ 94290us) raft_consensus.cc:1671] finished
0812 19:14:11.780582 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:11.780705 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":127686,"threads_started":1,"wal-append.queue_time_us":236746}
W20260812 19:14:11.867830 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 51088.6us
I20260812 19:14:11.814972 21203 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:11.742846 25023 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:11.814020 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:11.862862 22008 connection.cc:232] RPC call timeout handler was delayed by -1.6397524541243911s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19540000000000002s for a response
W20260812 19:14:11.877306 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 51904.2us
W20260812 19:14:11.870993 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 73917.8us
I20260812 19:14:11.771576 25133 raft_consensus.cc:482] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:11.884308 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414940a0231d4db9a5cba87d50c5f122" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:14:11.883611 21287 connection.cc:232] RPC call timeout handler was delayed by -1.0427031270228326s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2997s for a response
I20260812 19:14:11.884980 21241 raft_consensus.cc:3037] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:11.885447 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:11.886512 21287 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:11.886698 21236 raft_consensus.cc:2396] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260812 19:14:11.880913 21151 raft_consensus.cc:2454] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 9.
I20260812 19:14:11.890038 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:11.890753 21233 raft_consensus.cc:3037] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 8
I20260812 19:14:11.891458 21115 leader_election.cc:398] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Vote denied by peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579) with higher term. Message: Invalid argument: T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 8. Current term is 9.
I20260812 19:14:11.891789 21115 leader_election.cc:401] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260812 19:14:11.893844 21112 leader_election.cc:302] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:11.888824 21239 pending_rounds.cc:76] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa: Aborting all ops after (but not including) 7
I20260812 19:14:11.897809 21239 pending_rounds.cc:97] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260812 19:14:11.898149 21239 raft_consensus.cc:2866] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:14:11.900569 21287 leader_election.cc:302] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:11.894515 21241 raft_consensus.cc:2454] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 9.
I20260812 19:14:11.909961 21240 raft_consensus.cc:3032] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:14:11.910542 21240 raft_consensus.cc:729] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:11.911415 21240 consensus_queue.cc:260] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.791401 25010 leader_election.cc:288] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:11.795435 25202 raft_consensus.cc:729] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:11.918795 22056 raft_consensus.cc:2379] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 8: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
W20260812 19:14:11.876991 25311 log.cc:924] Time spent T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20260812 19:14:11.922298 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:11.928972 21237 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 7.
I20260812 19:14:11.869172 25215 raft_consensus.cc:3037] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Advancing to term 10
I20260812 19:14:11.863931 25149 consensus_queue.cc:1225] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer 63db10063a7d4eb2a9cc9dc952654706 log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260812 19:14:11.930126 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5090bb13ea4aa792b27b04a7a79688" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:11.938819 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:11.939388 21241 raft_consensus.cc:2354] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 8. Current term is 9.
I20260812 19:14:11.939492 21244 raft_consensus.cc:3037] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Advancing to term 9
W20260812 19:14:11.949484 22008 connection.cc:232] RPC call timeout handler was delayed by -1.3559286419767886s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
W20260812 19:14:11.950040 22008 connection.cc:232] RPC call timeout handler was delayed by -1.2207315689884126s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15400000000000003s for a response
I20260812 19:14:11.950738 22008 leader_election.cc:302] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:14:11.798983 25031 raft_consensus.cc:504] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.927544 25010 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:14:11.959156 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 144258us
W20260812 19:14:11.952747 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 55788.9us
I20260812 19:14:11.925138 24825 raft_consensus.cc:2726] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579) with higher term. Message: Invalid argument: T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 8. Current term is 9.
I20260812 19:14:11.881659 25256 consensus_queue.cc:1035] T 4c656db261bd45f8b421339ba53dd87f P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20260812 19:14:11.930317 24938 raft_consensus.cc:482] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:12.004173 21203 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:12.004269 22011 leader_election.cc:302] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:12.004092 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:12.078980 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:11.939246 25183 raft_consensus.cc:2726] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260812 19:14:11.931391 25174 raft_consensus.cc:2726] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260812 19:14:12.080197 25202 consensus_queue.cc:260] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:12.081537 25202 raft_consensus.cc:3037] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:14:12.013053 25010 raft_consensus.cc:504] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:12.012967 24606 consensus_queue.cc:260] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:11.917254 25133 raft_consensus.cc:504] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:12.088831 21236 raft_consensus.cc:1261] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260812 19:14:12.091329 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:14:12.064461 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 59431us
I20260812 19:14:11.877513 25199 raft_consensus.cc:504] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:12.079319 21241 raft_consensus.cc:2396] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260812 19:14:12.097506 21149 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 9.
I20260812 19:14:12.100418 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:12.097103 21233 raft_consensus.cc:2454] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:14:12.092270 21238 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:14:12.102409 21241 raft_consensus.cc:2379] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 10: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:14:12.005641 21243 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:14:12.104509 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:12.105551 21241 raft_consensus.cc:3037] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Advancing to term 6
W20260812 19:14:12.108788 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 158180us
I20260812 19:14:12.086320 24606 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:12.110159 22008 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:11.946110 25238 consensus_queue.cc:1046] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.093s
W20260812 19:14:12.037591 25237 consensus_meta.cc:287] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.573s user 0.012s sys 0.000s
W20260812 19:14:12.201179 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 92788.8us
W20260812 19:14:12.202252 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 89924.3us
I20260812 19:14:12.205394 21114 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:14:12.206933 21287 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.70.130:33807 timed out after 1.525s (SENT) [suppressed 18 similar messages]
I20260812 19:14:12.205314 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:14:12.207294 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 104967us
W20260812 19:14:12.211331 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 207981us
I20260812 19:14:12.212539 21203 leader_election.cc:302] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:12.212881 25265 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 7 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:12.215004 21241 raft_consensus.cc:2454] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
I20260812 19:14:12.215462 25200 consensus_queue.cc:1046] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260812 19:14:12.215741 25265 raft_consensus.cc:2781] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 19:14:12.216305 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:12.216598 25265 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:12.216794 25183 raft_consensus.cc:3032] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Stepping down as leader of term 7
I20260812 19:14:12.217324 25265 raft_consensus.cc:3037] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Advancing to term 5
I20260812 19:14:12.217607 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:12.232626 21114 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:11.970031 25225 consensus_queue.cc:1035] T e7e49dcb1cac44ee91df33a2abea1dd0 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.163s
W20260812 19:14:12.233121 21157 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51004 (request call id 474) took 1839 ms (client timeout 1893 ms). Trace:
I20260812 19:14:12.216629 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:12.238265 21319 raft_consensus.cc:2454] T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 9.
I20260812 19:14:12.107764 25312 raft_consensus.cc:482] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:12.230759 21327 raft_consensus.cc:1226] T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412 [term 5 LEADER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 4. Current term is 5. Ops: []
I20260812 19:14:12.241138 21114 leader_election.cc:398] T 2f5090bb13ea4aa792b27b04a7a79688 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Vote denied by peer d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807) with higher term. Message: Invalid argument: T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for earlier term 8. Current term is 9.
W20260812 19:14:12.237890 21157 rpcz_store.cc:269] 0812 19:14:10.393583 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:10.393713 (+ 130us) service_pool.cc:227] Handling call
0812 19:14:12.232889 (+1839176us) spinlock_profiling.cc:243] Waited 785 ms on lock 0x7b5400182754. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:12.233041 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1804570496}
I20260812 19:14:12.216032 25193 raft_consensus.cc:2726] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260812 19:14:12.244768 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 134700us
I20260812 19:14:12.256604 22008 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:12.215531 25303 raft_consensus.cc:2726] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260812 19:14:12.234962 24838 consensus_meta.cc:287] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 0.669s user 0.008s sys 0.007s
I20260812 19:14:12.119014 25284 leader_election.cc:288] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:12.194530 24991 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:12.252712 21241 raft_consensus.cc:2454] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:14:12.261636 21244 raft_consensus.cc:2454] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 9.
I20260812 19:14:12.089870 24938 raft_consensus.cc:504] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:12.231155 25259 leader_election.cc:288] T 85aa3ef35cc045a393fcbb13f15370a7 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:12.299170 25328 raft_consensus.cc:2781] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader election won for term 9
I20260812 19:14:12.270977 21156 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 9.
I20260812 19:14:12.233501 25237 raft_consensus.cc:2726] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 19:14:12.217407 25183 raft_consensus.cc:729] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:12.285307 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:12.202759 25031 leader_election.cc:288] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:12.287552 21203 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412
W20260812 19:14:12.418463 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 131051us
I20260812 19:14:12.295325 21318 raft_consensus.cc:1226] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
W20260812 19:14:12.203167 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 288554us
I20260812 19:14:12.304806 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:12.306705 21200 leader_election.cc:302] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:12.240777 24844 raft_consensus.cc:2726] T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:14:12.312188 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcaba8d2e2c42b48d7fa20deee5b73f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:12.242751 25320 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:12.354873 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:12.352679 24991 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:12.279659 25301 raft_consensus.cc:2781] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Leader election won for term 9
I20260812 19:14:12.350556 24971 leader_election.cc:288] T 4269140f589f4a1fa81fed02bfd8f2bd P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:12.357952 25023 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:12.276711 25151 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:12.318068 25236 consensus_meta.cc:287] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.534s user 0.012s sys 0.000s
I20260812 19:14:12.369578 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
W20260812 19:14:12.436600 21155 consensus_meta.cc:287] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.705s user 0.010s sys 0.002s
I20260812 19:14:12.437057 21155 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8.
I20260812 19:14:12.374147 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:12.404575 25161 leader_election.cc:288] T 9548455d4eee4c74a47bf4b912e5740d P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:12.270948 25244 leader_election.cc:288] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:12.357141 25153 raft_consensus.cc:2781] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:12.451752 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:12.452682 21244 raft_consensus.cc:2379] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 8: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:12.249547 24997 consensus_queue.cc:1046] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.109s
I20260812 19:14:12.270737 25312 raft_consensus.cc:504] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:12.413352 21319 raft_consensus.cc:3037] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Advancing to term 6
I20260812 19:14:12.462949 24997 raft_consensus.cc:3032] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 7 LEADER]: Stepping down as leader of term 7
I20260812 19:14:12.463413 24997 raft_consensus.cc:729] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
I20260812 19:14:12.464329 24997 consensus_queue.cc:260] T 932fac0d61e94da3a0f9c11ae2f597c3 P 93578836968649f6bfcaecc709871412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:12.425583 25303 raft_consensus.cc:2741] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260812 19:14:12.365883 25010 leader_election.cc:288] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:12.466204 25303 consensus_queue.cc:1046] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260812 19:14:12.422349 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 115842us
I20260812 19:14:12.466069 25183 consensus_queue.cc:260] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:12.467532 25303 raft_consensus.cc:3032] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:14:12.467828 25183 raft_consensus.cc:3037] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:12.467969 25303 raft_consensus.cc:729] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:12.425561 25330 raft_consensus.cc:2726] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:14:12.469007 25303 consensus_queue.cc:260] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:12.309736 24838 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:12.372711 21025 maintenance_manager.cc:419] P 61e1609653694f899be42137777c67c1: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033403
W20260812 19:14:12.360327 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 103843us
I20260812 19:14:12.471726 24938 leader_election.cc:288] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:12.472307 21322 raft_consensus.cc:1226] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 7. Current term is 9. Ops: []
I20260812 19:14:12.472808 24843 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:12.452219 25153 raft_consensus.cc:482] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:12.473624 25153 raft_consensus.cc:3037] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:12.473511 24843 raft_consensus.cc:504] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } attrs { replace: true } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:12.417014 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 165987us
W20260812 19:14:12.474857 25318 log.cc:924] Time spent T 32332933db0a4070bbf6fbabe17e716d P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.243s user 0.001s sys 0.000s
I20260812 19:14:12.426433 21241 raft_consensus.cc:2379] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 6: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:14:12.475371 21114 leader_election.cc:302] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:12.429926 21158 raft_consensus.cc:1226] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
I20260812 19:14:12.476311 25332 raft_consensus.cc:2781] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader election won for term 8
I20260812 19:14:12.476433 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:12.476903 25332 raft_consensus.cc:686] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 8 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:12.248314 25238 raft_consensus.cc:3032] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Stepping down as leader of term 7
I20260812 19:14:12.489773 21287 leader_election.cc:398] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 8. Current term is 9.
I20260812 19:14:12.490216 21287 leader_election.cc:401] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:14:12.437776 22047 raft_consensus.cc:2454] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
W20260812 19:14:12.439694 25319 log.cc:924] Time spent T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.323s user 0.000s sys 0.000s
I20260812 19:14:12.427448 25133 leader_election.cc:288] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:12.440346 25317 log.cc:924] Time spent T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.352s user 0.001s sys 0.000s
W20260812 19:14:12.441423 25321 log.cc:924] Time spent T 1c678904ece144c29ee1c70ff4568969 P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260812 19:14:12.442308 21238 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8.
I20260812 19:14:12.449756 21323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 2163) took 2343 ms. Trace:
I20260812 19:14:12.411448 25328 raft_consensus.cc:686] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [term 9 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:12.453964 21203 leader_election.cc:302] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:12.501020 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:12.501818 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 47957.5us
I20260812 19:14:12.502991 21203 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:12.451659 24996 raft_consensus.cc:2726] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:12.477774 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:12.448709 25301 raft_consensus.cc:686] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 LEADER]: Becoming Leader. State: Replica: 63db10063a7d4eb2a9cc9dc952654706, State: Running, Role: LEADER
I20260812 19:14:12.451939 24971 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:12.479851 22008 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:12.478114 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:12.519569 21244 raft_consensus.cc:3037] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Advancing to term 9
W20260812 19:14:12.522071 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 518410us
I20260812 19:14:12.486485 21319 raft_consensus.cc:2454] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:14:12.425822 25193 raft_consensus.cc:2781] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260812 19:14:12.491927 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:12.588428 21327 raft_consensus.cc:2379] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 9: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:14:12.440201 25227 raft_consensus.cc:3032] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 9 LEADER]: Stepping down as leader of term 9
I20260812 19:14:12.498075 21288 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:14:12.505733 21203 leader_election.cc:302] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:12.501739 21328 raft_consensus.cc:2454] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:14:12.500648 21323 rpcz_store.cc:276] 0812 19:14:10.106201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:10.123748 (+ 17547us) service_pool.cc:227] Handling call
0812 19:14:10.124014 (+ 266us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:10.124187 (+ 173us) raft_consensus.cc:1520] Early marking committed up to index 7
0812 19:14:10.159107 (+ 34920us) log.cc:842] Serialized 14 byte log entry
0812 19:14:12.444032 (+2284925us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b180046abc0. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d973582c 00007f52d6b6c9dc 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b89e22 00007f52d97275bb 00007f52d972d063 00007f52d972d57b
0812 19:14:12.449445 (+ 5413us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:12.449502 (+ 57us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:14:12.449537 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:12.449595 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:12.449704 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5143,"prepare.queue_time_us":35231,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":4908,"raft.queue_time_us":18411,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"spinlock_wait_cycles":27679616,"thread_start_us":1128319,"threads_started":1,"wal-append.queue_time_us":1126182}
W20260812 19:14:12.602378 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 96732.6us
I20260812 19:14:12.430334 25167 consensus_queue.cc:1046] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.060s
I20260812 19:14:12.515218 21241 raft_consensus.cc:2454] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:14:12.604269 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:12.426916 21112 leader_election.cc:302] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:14:12.605747 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 178930us
I20260812 19:14:12.606379 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:12.516494 21287 leader_election.cc:302] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:12.502750 25330 consensus_queue.cc:1035] T 8c13f33db56241a7938cfe52798e77a2 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260812 19:14:12.489159 25238 raft_consensus.cc:729] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:12.437816 25320 raft_consensus.cc:504] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:12.500335 25303 raft_consensus.cc:3037] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:12.522533 25337 raft_consensus.cc:2726] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 8. Current term is 9.
I20260812 19:14:12.515156 25133 raft_consensus.cc:2726] T 414940a0231d4db9a5cba87d50c5f122 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260812 19:14:12.478389 25200 consensus_queue.cc:1159] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f741102bfe1441d1b453e5a838649059" responder_term: 7 status { last_received { term: 7 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 7 index: 4 } }
I20260812 19:14:12.488096 24606 consensus_queue.cc:1046] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260812 19:14:12.444386 25236 raft_consensus.cc:504] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:12.534242 21204 leader_election.cc:302] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:14:12.514971 25010 raft_consensus.cc:2726] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260812 19:14:12.559554 25331 raft_consensus.cc:2726] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:14:12.514063 25328 consensus_queue.cc:237] T 414940a0231d4db9a5cba87d50c5f122 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:12.739229 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aa3ef35cc045a393fcbb13f15370a7" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
W20260812 19:14:12.553820 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 74076us
I20260812 19:14:12.521585 25260 raft_consensus.cc:2726] T ae9be4809de24d32bdd888c6f9511d0f P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260812 19:14:12.517601 24838 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:12.558483 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadd5203f7ca4c04b02bd28bea6b591c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:14:12.556134 21243 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57682 (request call id 399) took 1995 ms (client timeout 1950 ms). Trace:
W20260812 19:14:12.592799 22011 connection.cc:232] RPC call timeout handler was delayed by -1.2166769879404455s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.195s for a response
I20260812 19:14:12.682328 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:12.599989 21244 raft_consensus.cc:2454] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 9.
I20260812 19:14:12.615761 21327 raft_consensus.cc:1261] T 4f9f5154fdef497797568cd2e10742be P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260812 19:14:12.471833 25325 consensus_queue.cc:1046] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260812 19:14:12.604991 21241 raft_consensus.cc:2362] T 49361681c5c1421594d10cd52061ec44 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7. Re-sending same reply.
I20260812 19:14:12.751477 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:14:12.620515 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 104080us
I20260812 19:14:12.604404 21203 leader_election.cc:398] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 7. Current term is 8.
I20260812 19:14:12.752593 21287 leader_election.cc:302] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:12.629583 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:12.754029 21328 raft_consensus.cc:2396] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260812 19:14:12.635074 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:12.525669 24971 raft_consensus.cc:504] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:12.601079 25227 raft_consensus.cc:729] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:12.640679 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49361681c5c1421594d10cd52061ec44" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:12.770314 21329 raft_consensus.cc:3037] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:12.733879 25320 leader_election.cc:288] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:12.589563 25332 consensus_queue.cc:237] T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:12.730720 25290 raft_consensus.cc:3032] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:14:12.731258 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
W20260812 19:14:12.748288 21243 rpcz_store.cc:269] 0812 19:14:10.560691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:10.644413 (+ 83722us) service_pool.cc:227] Handling call
0812 19:14:12.555880 (+1911467us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54002c42d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:12.556043 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3039355136}
I20260812 19:14:12.789475 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:12.790375 21243 raft_consensus.cc:2379] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 9: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:14:12.791203 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:12.791936 21243 raft_consensus.cc:2379] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 6: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:12.751619 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc50dccc6345495aaec42b51dc30d299" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
W20260812 19:14:12.620409 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 122456us
I20260812 19:14:12.745800 24843 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:12.631748 21149 raft_consensus.cc:1226] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 7. Current term is 8. Ops: []
I20260812 19:14:12.800781 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:12.802296 21149 raft_consensus.cc:1261] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260812 19:14:12.754993 21323 raft_consensus.cc:2396] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260812 19:14:12.587853 25193 raft_consensus.cc:482] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting leader election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
W20260812 19:14:12.627916 25191 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.177s user 0.008s sys 0.006s
I20260812 19:14:12.625254 25237 raft_consensus.cc:2781] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader election won for term 9
I20260812 19:14:12.767560 21327 raft_consensus.cc:1261] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Refusing update from remote peer f741102bfe1441d1b453e5a838649059: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260812 19:14:12.769632 25161 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:12.810133 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:12.810904 21327 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 9.
I20260812 19:14:12.811980 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:12.720320 25215 consensus_meta.cc:287] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.751s user 0.011s sys 0.002s
I20260812 19:14:12.777089 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad64c74a7de6449c8152bc0e093a59db" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:12.777463 22053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:14:12.735116 24993 consensus_meta.cc:287] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 0.959s user 0.011s sys 0.003s
I20260812 19:14:12.778995 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:14:12.701777 25198 consensus_meta.cc:287] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.171s user 0.012s sys 0.003s
W20260812 19:14:12.784392 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 250268us
I20260812 19:14:12.788630 25290 raft_consensus.cc:729] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:12.703495 25337 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:12.730424 25338 raft_consensus.cc:2781] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:14:12.729866 25358 raft_consensus.cc:2726] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:12.752904 21203 leader_election.cc:401] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260812 19:14:12.728695 25344 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:12.795274 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:12.796396 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a915c8829aa4e1fae0110e009af0416" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:12.887722 21241 raft_consensus.cc:2354] T 8a915c8829aa4e1fae0110e009af0416 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for earlier term 8. Current term is 9.
I20260812 19:14:12.797416 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:12.798494 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6309c277c104b23a58d4824d764d397" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:12.803910 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab2f57c1cbc44b6976aeb480e73ad2c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:12.890089 21236 raft_consensus.cc:2454] T cab2f57c1cbc44b6976aeb480e73ad2c P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7.
I20260812 19:14:12.891371 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2117fe1b5b8c42758a313da21e01a17c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:12.892396 21236 raft_consensus.cc:2379] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 10: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:14:12.893414 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:12.804003 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4269140f589f4a1fa81fed02bfd8f2bd" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:12.769106 25023 raft_consensus.cc:2726] T 8c13f33db56241a7938cfe52798e77a2 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 19:14:12.763929 25324 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:12.805332 21329 raft_consensus.cc:2454] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 8.
I20260812 19:14:12.896903 25324 raft_consensus.cc:504] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:12.897969 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bb026b4ce64ee6beb14a67a87b06af" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:12.807561 21149 raft_consensus.cc:1261] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260812 19:14:12.899231 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d22e3e7455b4eb0806e88ab3f5f4b3b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:12.849535 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:12.899866 21241 raft_consensus.cc:2362] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 93578836968649f6bfcaecc709871412 in term 7. Re-sending same reply.
I20260812 19:14:12.851914 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392684a63faf471c902391da6a7b0308" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:12.850055 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:12.781361 25331 raft_consensus.cc:482] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:14:12.854445 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 109851us
W20260812 19:14:12.809051 25191 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.372s user 0.008s sys 0.016s
I20260812 19:14:12.800552 25031 raft_consensus.cc:482] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:12.806155 25330 raft_consensus.cc:482] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:12.857352 21204 leader_election.cc:302] T 45e03c303e40432e8f5da639f7edff82 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:12.876763 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784d71ae37454b3a9e8023f268428468" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:12.810743 25193 raft_consensus.cc:3037] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Advancing to term 10
I20260812 19:14:12.801103 25045 raft_consensus.cc:2726] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:12.772858 25153 raft_consensus.cc:504] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:12.897778 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:12.909157 25031 raft_consensus.cc:504] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:12.910384 21152 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:12.909253 25331 raft_consensus.cc:504] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:12.910965 25153 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:12.911998 21329 raft_consensus.cc:2454] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 3.
W20260812 19:14:12.913113 22008 connection.cc:232] RPC call timeout handler was delayed by -1.065502185956575s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2242s for a response
I20260812 19:14:12.913069 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:12.913899 21329 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 9.
I20260812 19:14:12.915233 25330 raft_consensus.cc:504] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } attrs { replace: true } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:12.915350 25331 leader_election.cc:288] T eadd5203f7ca4c04b02bd28bea6b591c P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:12.918423 21204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 61156.1us
I20260812 19:14:12.909897 21323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:12.919268 21204 leader_election.cc:302] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:12.920246 25045 raft_consensus.cc:2726] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:14:12.920538 21152 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 9.
W20260812 19:14:12.923684 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 319442us
I20260812 19:14:12.923887 21204 leader_election.cc:302] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:14:12.915350 21156 raft_consensus.cc:2379] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 6: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:12.850483 21327 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:14:12.850983 21287 leader_election.cc:302] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:12.928304 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:12.929052 21327 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 6.
I20260812 19:14:12.926841 21204 leader_election.cc:302] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059
I20260812 19:14:12.853181 24991 leader_election.cc:288] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:12.882985 25354 log.cc:924] Time spent T a8161580b52441ab9a17a0e1597f243b P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
I20260812 19:14:12.899612 25324 leader_election.cc:288] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:12.941955 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:12.900811 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9be4809de24d32bdd888c6f9511d0f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:12.879915 25338 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:12.943810 21327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:12.923421 25151 raft_consensus.cc:482] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:12.951668 21241 raft_consensus.cc:2362] T ae9be4809de24d32bdd888c6f9511d0f P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6. Re-sending same reply.
I20260812 19:14:12.911610 25031 leader_election.cc:288] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:12.956674 21327 raft_consensus.cc:2379] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 7: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:14:12.772560 25238 consensus_queue.cc:260] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:12.719704 25301 consensus_queue.cc:237] T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:12.861722 24993 raft_consensus.cc:504] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:12.950744 25325 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:12.961726 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8161580b52441ab9a17a0e1597f243b" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:12.961838 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:12.942878 25331 raft_consensus.cc:482] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:14:12.958195 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 163313us
W20260812 19:14:12.941874 25246 consensus_meta.cc:287] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.235s user 0.015s sys 0.000s
W20260812 19:14:12.967887 21151 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51004 (request call id 477) took 1716 ms (client timeout 1984 ms). Trace:
W20260812 19:14:12.958240 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 171117us
I20260812 19:14:12.967931 21150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 2148) took 1351 ms. Trace:
I20260812 19:14:13.006832 21150 rpcz_store.cc:276] 0812 19:14:11.616393 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.637909 (+ 21516us) service_pool.cc:227] Handling call
0812 19:14:11.638252 (+ 343us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:11.859569 (+221317us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:12.967691 (+1108122us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b540019d7d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:12.967877 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":268609408}
I20260812 19:14:12.941589 25362 raft_consensus.cc:2726] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551) with higher term. Message: Invalid argument: T 9548455d4eee4c74a47bf4b912e5740d P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa for earlier term 7. Current term is 8.
I20260812 19:14:12.959772 25338 raft_consensus.cc:3037] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:12.983215 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce6040ab9fc41539e7fc09b3f9d05dd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:12.853961 25215 raft_consensus.cc:504] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:12.861544 25161 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:12.946513 25358 raft_consensus.cc:2726] T ad64c74a7de6449c8152bc0e093a59db P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260812 19:14:12.955845 25265 consensus_meta.cc:287] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.720s user 0.008s sys 0.005s
I20260812 19:14:13.022994 22045 raft_consensus.cc:1261] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260812 19:14:13.023272 25265 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:13.024276 25238 raft_consensus.cc:3037] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:13.025059 25368 raft_consensus.cc:3037] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:13.026710 25265 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 5 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:13.028362 25161 leader_election.cc:288] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:12.996306 21200 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
W20260812 19:14:13.029393 25202 consensus_meta.cc:287] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.947s user 0.015s sys 0.001s
I20260812 19:14:12.943063 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:13.042961 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 46775us
I20260812 19:14:13.043696 21329 raft_consensus.cc:2379] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 6: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:14:13.043923 21200 leader_election.cc:302] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:12.947382 25237 raft_consensus.cc:686] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [term 9 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
I20260812 19:14:13.001092 22051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:13.001335 22047 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 9.
I20260812 19:14:12.954195 24838 raft_consensus.cc:482] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:12.954195 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:12.963986 24606 raft_consensus.cc:2726] T c6309c277c104b23a58d4824d764d397 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 19:14:12.860481 25227 consensus_queue.cc:260] T 2117fe1b5b8c42758a313da21e01a17c P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:14:12.969611 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 118691us
I20260812 19:14:13.074429 21287 leader_election.cc:302] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:12.970849 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:13.075721 21156 raft_consensus.cc:2454] T 65db14a4838d40218947e8bd3420e892 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 9.
W20260812 19:14:13.006038 21151 rpcz_store.cc:269] 0812 19:14:11.251229 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.263202 (+ 11973us) service_pool.cc:227] Handling call
0812 19:14:12.967691 (+1704489us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b54001a3454. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:12.967822 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":592914688}
I20260812 19:14:13.014602 21154 raft_consensus.cc:2454] T 9ce6040ab9fc41539e7fc09b3f9d05dd P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 8.
I20260812 19:14:13.015408 21155 raft_consensus.cc:1226] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 7. Current term is 8. Ops: []
I20260812 19:14:12.990924 25045 raft_consensus.cc:482] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:14:13.018911 25343 log.cc:924] Time spent T 38dbcfe527fe4701863d72b9bb480ccf P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
W20260812 19:14:13.020511 25351 log.cc:924] Time spent T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20260812 19:14:12.861243 25323 consensus_queue.cc:1046] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.095s
I20260812 19:14:13.029304 25215 leader_election.cc:288] T e7e49dcb1cac44ee91df33a2abea1dd0 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 10 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:13.053493 22047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b730f70f25e440e68595cc248b40ee98" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:13.050590 22051 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:14:13.060640 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff4931ad1bb4657aa3650a15b43ddcd" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:13.072475 25202 raft_consensus.cc:2781] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:14:12.923707 25337 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:13.071669 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fac0d61e94da3a0f9c11ae2f597c3" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:12.972447 25151 raft_consensus.cc:504] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:13.081198 22011 leader_election.cc:302] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
W20260812 19:14:13.153389 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 72260.8us
I20260812 19:14:13.082140 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:13.081876 25236 leader_election.cc:288] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 election: Requested vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:13.084177 21200 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:13.081043 21237 raft_consensus.cc:2454] T 32332933db0a4070bbf6fbabe17e716d P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
I20260812 19:14:13.093375 21112 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:13.047899 24971 leader_election.cc:288] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:13.058375 24843 raft_consensus.cc:2781] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader election won for term 6
I20260812 19:14:12.943336 25344 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:13.013051 25246 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:13.125953 22047 raft_consensus.cc:2454] T b730f70f25e440e68595cc248b40ee98 P 63db10063a7d4eb2a9cc9dc952654706 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 3.
I20260812 19:14:13.064970 22048 raft_consensus.cc:1261] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Refusing update from remote peer 93578836968649f6bfcaecc709871412: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260812 19:14:13.126763 21149 raft_consensus.cc:2454] T aff4931ad1bb4657aa3650a15b43ddcd P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 6.
I20260812 19:14:13.100507 25045 raft_consensus.cc:504] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:13.149398 21236 raft_consensus.cc:2379] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 8: Already voted for candidate d1e1da0b9585460e8c365bf55679d4fa in this term.
I20260812 19:14:13.014560 25331 raft_consensus.cc:504] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:13.160065 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 75995.8us
I20260812 19:14:13.182350 21200 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:13.171454 25320 raft_consensus.cc:2781] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260812 19:14:13.171811 25265 raft_consensus.cc:2781] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 19:14:13.172545 25374 consensus_queue.cc:1035] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260812 19:14:13.077438 24838 raft_consensus.cc:504] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:13.178023 21288 leader_election.cc:302] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412; no voters:
I20260812 19:14:13.004781 25325 raft_consensus.cc:504] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:13.151166 25202 raft_consensus.cc:482] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:13.172859 25044 leader_election.cc:288] T 5d22e3e7455b4eb0806e88ab3f5f4b3b P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:13.247876 21157 raft_consensus.cc:2379] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 7: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:13.233047 21328 raft_consensus.cc:2454] T 4c656db261bd45f8b421339ba53dd87f P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 8.
I20260812 19:14:13.211809 25236 raft_consensus.cc:482] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:13.252682 25236 raft_consensus.cc:504] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:13.209152 24843 raft_consensus.cc:686] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:14:13.254000 25183 consensus_meta.cc:287] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 0.786s user 0.012s sys 0.003s
I20260812 19:14:13.209481 25320 raft_consensus.cc:482] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:13.254954 25320 raft_consensus.cc:3037] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa [term 10 FOLLOWER]: Advancing to term 11
I20260812 19:14:13.254750 24843 consensus_queue.cc:237] T e06f074e35674592bdc8360835fd2661 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:13.254786 21244 raft_consensus.cc:2454] T 4269140f589f4a1fa81fed02bfd8f2bd P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8.
W20260812 19:14:13.231897 21115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 49571.8us
I20260812 19:14:13.211412 25237 consensus_queue.cc:237] T 8a915c8829aa4e1fae0110e009af0416 P f741102bfe1441d1b453e5a838649059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:13.237304 25362 raft_consensus.cc:2781] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:14:13.237564 25045 leader_election.cc:288] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:13.263609 25362 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:13.264185 25362 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
W20260812 19:14:13.238225 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 60291.8us
I20260812 19:14:13.239315 24971 raft_consensus.cc:2781] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 19:14:13.240697 25344 leader_election.cc:288] T 41fc49d178354e629e97311498b381ec P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:13.242859 25357 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:13.243116 25365 raft_consensus.cc:2726] T e06f074e35674592bdc8360835fd2661 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260812 19:14:13.243727 25335 log.cc:924] Time spent T 0569d19f4a06475c84bdc61b1c5f549f P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260812 19:14:13.244441 25151 leader_election.cc:288] T 32332933db0a4070bbf6fbabe17e716d P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:13.231722 24606 raft_consensus.cc:482] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:13.226404 25265 raft_consensus.cc:482] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:13.007498 25290 consensus_queue.cc:260] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:13.147751 25370 consensus_queue.cc:1035] T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.096s
I20260812 19:14:13.237073 25329 raft_consensus.cc:2726] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260812 19:14:13.318215 21324 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1954) took 2851 ms (client timeout 1853 ms). Trace:
I20260812 19:14:13.250591 25202 raft_consensus.cc:3037] T 4f9f5154fdef497797568cd2e10742be P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:13.307793 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:13.308046 24606 raft_consensus.cc:504] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:13.309193 25265 raft_consensus.cc:3037] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:13.315001 25290 raft_consensus.cc:3037] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
W20260812 19:14:13.315744 25313 consensus_queue.cc:1159] T d58350d75f2f4a61b79148bd023ff99c P f741102bfe1441d1b453e5a838649059 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "63db10063a7d4eb2a9cc9dc952654706" responder_term: 6 status { last_received { term: 6 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 6 index: 6 } }
I20260812 19:14:13.319063 21320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1957) took 2177 ms. Trace:
I20260812 19:14:13.282424 24971 raft_consensus.cc:482] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:14:13.786433 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 712038us
I20260812 19:14:13.788049 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:13.282127 25374 raft_consensus.cc:482] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:13.194531 25330 leader_election.cc:288] T 65db14a4838d40218947e8bd3420e892 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:13.792955 21153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54796 (request call id 1760) took 2102 ms. Trace:
I20260812 19:14:13.802807 21153 rpcz_store.cc:276] 0812 19:14:11.690715 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.863715 (+173000us) service_pool.cc:227] Handling call
0812 19:14:11.909836 (+ 46121us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:11.932368 (+ 22532us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:11.932399 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:12.442147 (+509748us) spinlock_profiling.cc:243] Waited 7.71 ms on lock 0x7b2c0006b848. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d8495e11 00007f52d84a5442 00007f52d9ae1c11 00007f52d9aa658a 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772
0812 19:14:12.442679 (+ 532us) log.cc:842] Serialized 26 byte log entry
0812 19:14:12.764810 (+322131us) raft_consensus.cc:1631] Marking committed up to 8
0812 19:14:13.126256 (+361446us) log.cc:842] Serialized 14 byte log entry
0812 19:14:13.786100 (+659844us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:13.786212 (+ 112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:13.786277 (+ 65us) raft_consensus.cc:1671] finished
0812 19:14:13.792685 (+ 6408us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:13.792886 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45720,"prepare.queue_time_us":416180,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":38477,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":32148,"spinlock_wait_cycles":17723648,"thread_start_us":625969,"threads_started":2,"wal-append.queue_time_us":501950}
I20260812 19:14:13.792795 22047 raft_consensus.cc:2932] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Committing config change with OpId 5.5: config changed from index 2 to 5, 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } attrs { replace: true } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } } }
I20260812 19:14:13.811925 22048 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741102bfe1441d1b453e5a838649059 in term 5.
W20260812 19:14:13.810032 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 716761us
W20260812 19:14:13.812624 21324 rpcz_store.cc:269] 0812 19:14:10.467359 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:10.472146 (+ 4787us) service_pool.cc:227] Handling call
0812 19:14:13.317943 (+2845797us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400223dd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:13.318095 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2817248000}
W20260812 19:14:13.493933 21238 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1670) took 1486 ms (client timeout 1596 ms). Trace:
I20260812 19:14:13.809289 21155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51004 (request call id 478) took 1282 ms. Trace:
W20260812 19:14:13.820036 21238 rpcz_store.cc:269] 0812 19:14:12.007867 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:12.102334 (+ 94467us) service_pool.cc:227] Handling call
0812 19:14:13.493632 (+1391298us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b540014ffd4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:13.493828 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":539291392}
I20260812 19:14:13.821323 22055 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40614 (request call id 615) took 1379 ms. Trace:
W20260812 19:14:13.381748 22056 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40624 (request call id 712) took 1627 ms (client timeout 1998 ms). Trace:
W20260812 19:14:13.823827 22056 rpcz_store.cc:269] 0812 19:14:11.753968 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.755547 (+ 1579us) service_pool.cc:227] Handling call
0812 19:14:13.381385 (+1625838us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b54004538d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:13.381641 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":325011840}
I20260812 19:14:13.551196 21243 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 8.
I20260812 19:14:13.819335 21112 leader_election.cc:302] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:13.336134 25135 leader_election.cc:288] T c31c6a2accba4d29ba096500b9d1a911 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:13.319933 24993 leader_election.cc:288] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:13.787161 25246 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 election: Requested vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:13.383253 25325 leader_election.cc:288] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:13.319585 25331 leader_election.cc:288] T 45e03c303e40432e8f5da639f7edff82 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
W20260812 19:14:13.321913 25300 log.cc:924] Time spent T 4f9f5154fdef497797568cd2e10742be P f741102bfe1441d1b453e5a838649059: Append to log took a long time: real 1.830s user 0.000s sys 0.005s
I20260812 19:14:13.845525 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06f074e35674592bdc8360835fd2661" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:13.833660 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9ea4c9e414d1093d28b6c91e19ff0" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:14:13.844794 21321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51568 (request call id 1960) took 2698 ms (client timeout 1730 ms). Trace:
I20260812 19:14:13.832496 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f5154fdef497797568cd2e10742be" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:13.820472 21155 rpcz_store.cc:276] 0812 19:14:12.527401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:12.527571 (+ 170us) service_pool.cc:227] Handling call
0812 19:14:12.527925 (+ 354us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:13.081696 (+553771us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:13.314690 (+232994us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b54001a3954. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:13.809199 (+494509us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1115676032}
I20260812 19:14:13.845906 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405717c8c644382be14b4f6cd0a3bd7" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:13.818912 21320 rpcz_store.cc:276] 0812 19:14:11.141272 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.166859 (+ 25587us) service_pool.cc:227] Handling call
0812 19:14:13.318860 (+2152001us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b5400312ed4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:13.318990 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":557949824}
W20260812 19:14:13.846294 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 58730.7us
I20260812 19:14:13.627852 25236 leader_election.cc:288] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:13.849571 21154 pending_rounds.cc:76] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059: Aborting all ops after (but not including) 4
I20260812 19:14:13.855857 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:13.858623 21112 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:13.445334 25199 leader_election.cc:288] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:13.290350 25357 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:13.822651 22055 rpcz_store.cc:276] 0812 19:14:12.442272 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:12.444966 (+ 2694us) service_pool.cc:227] Handling call
0812 19:14:12.445237 (+ 271us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:12.470566 (+ 25329us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:12.470595 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:12.470754 (+ 159us) raft_consensus.cc:1631] Marking committed up to 8
0812 19:14:12.470809 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:13.385734 (+914925us) spinlock_profiling.cc:243] Waited 25.1 ms on lock 0x7b540061fb54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:13.385815 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:13.821240 (+435425us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57739648}
I20260812 19:14:13.860951 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9548455d4eee4c74a47bf4b912e5740d" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:14:14.032074 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 173562us
I20260812 19:14:13.866189 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc64ef202b904bcf809760ddfcd1f140" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:14.033164 22050 raft_consensus.cc:2379] T dc64ef202b904bcf809760ddfcd1f140 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f741102bfe1441d1b453e5a838649059 in current term 7: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:14:13.890492 22048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7825e8ab0b2e4d7abbb7dd19c9e3ad69" candidate_uuid: "f741102bfe1441d1b453e5a838649059" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706"
I20260812 19:14:14.033733 21287 leader_election.cc:302] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:14:13.983067 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:14.035192 21287 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters:
I20260812 19:14:14.035348 21151 raft_consensus.cc:2454] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:14:14.032949 21112 leader_election.cc:302] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:14.031360 21154 pending_rounds.cc:97] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260812 19:14:14.038491 21154 raft_consensus.cc:2866] T 0569d19f4a06475c84bdc61b1c5f549f P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 19:14:13.450016 25322 raft_consensus.cc:3032] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 9 LEADER]: Stepping down as leader of term 9
I20260812 19:14:13.628309 25260 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
I20260812 19:14:13.646412 25215 raft_consensus.cc:482] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 19:14:14.031632 21321 rpcz_store.cc:269] 0812 19:14:11.146789 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.146954 (+ 165us) service_pool.cc:227] Handling call
0812 19:14:13.844531 (+2697577us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54002959d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:13.844698 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3245923072}
W20260812 19:14:13.826493 25306 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.747s user 0.003s sys 0.012s
I20260812 19:14:14.032375 21236 raft_consensus.cc:2379] T 9548455d4eee4c74a47bf4b912e5740d P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 63db10063a7d4eb2a9cc9dc952654706 in current term 8: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:14.040767 22011 leader_election.cc:302] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:14.056517 21236 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57682 (request call id 409) took 1062 ms. Trace:
I20260812 19:14:14.057030 21236 rpcz_store.cc:276] 0812 19:14:12.993732 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:13.852837 (+859105us) service_pool.cc:227] Handling call
0812 19:14:14.056451 (+203614us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:13.935514 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c8327f92104d69b0af3f6e7bf28d17" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:14.036317 21287 leader_election.cc:302] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:14:13.379626 25184 raft_consensus.cc:3032] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Stepping down as leader of term 6
I20260812 19:14:14.097316 25184 raft_consensus.cc:729] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:14.058830 25385 raft_consensus.cc:2781] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:14.098011 25385 raft_consensus.cc:482] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:14.098448 25385 raft_consensus.cc:3037] T c31c6a2accba4d29ba096500b9d1a911 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:14.098950 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:14.046341 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412"
I20260812 19:14:14.099892 21321 raft_consensus.cc:3037] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:14.100481 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059"
I20260812 19:14:14.102036 21153 raft_consensus.cc:1226] T 38dbcfe527fe4701863d72b9bb480ccf P f741102bfe1441d1b453e5a838649059 [term 10 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 9. Current term is 10. Ops: []
I20260812 19:14:14.104723 25389 consensus_queue.cc:1035] T a070b29d02d24ef4b36656fcac6d68d9 P 93578836968649f6bfcaecc709871412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260812 19:14:14.107368 25312 leader_election.cc:288] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:14.108880 21158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54790 (request call id 2137) took 2729 ms. Trace:
I20260812 19:14:14.109527 21321 raft_consensus.cc:2454] T 7dfc261e3ffb48ddaa432f57bf7aad22 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 9.
I20260812 19:14:14.109455 21158 rpcz_store.cc:276] 0812 19:14:11.379232 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.380565 (+ 1333us) service_pool.cc:227] Handling call
0812 19:14:11.429775 (+ 49210us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:12.476602 (+1046827us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:14.108421 (+1631819us) spinlock_profiling.cc:243] Waited 995 ms on lock 0x7b5400182c54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:14.108740 (+ 319us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2287439616}
I20260812 19:14:14.110858 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:14.111472 21152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54796 (request call id 1768) took 1324 ms (client timeout 1752 ms). Trace:
I20260812 19:14:14.111698 24606 leader_election.cc:288] T d6521960ce6040af919a9e227d2211ec P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
W20260812 19:14:14.112063 21152 rpcz_store.cc:269] 0812 19:14:12.790094 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:12.795970 (+ 5876us) service_pool.cc:227] Handling call
0812 19:14:14.111211 (+1315241us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b54000c2e54. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:14.111383 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23374208}
I20260812 19:14:14.048529 21203 leader_election.cc:302] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:14.116245 25390 raft_consensus.cc:2781] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:14.117125 25138 raft_consensus.cc:2726] T 5405717c8c644382be14b4f6cd0a3bd7 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260812 19:14:14.117254 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 68838us
I20260812 19:14:14.111647 21321 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 8.
I20260812 19:14:14.120247 21153 raft_consensus.cc:1226] T 83c3d64ee3474fd0b1c7fd42a6cd30d3 P f741102bfe1441d1b453e5a838649059 [term 6 LEADER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 5. Current term is 6. Ops: []
I20260812 19:14:14.120528 25133 raft_consensus.cc:482] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:14.121686 21153 raft_consensus.cc:1226] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 8. Current term is 9. Ops: []
I20260812 19:14:14.121255 25133 raft_consensus.cc:504] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:14.120595 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c678904ece144c29ee1c70ff4568969" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:14.123467 25384 raft_consensus.cc:482] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:14.124312 25384 raft_consensus.cc:504] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:14.125308 21329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 2192) took 1331 ms. Trace:
I20260812 19:14:14.123868 21321 raft_consensus.cc:2454] T 1c678904ece144c29ee1c70ff4568969 P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 7.
I20260812 19:14:14.127569 21321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51552 (request call id 2195) took 1119 ms. Trace:
I20260812 19:14:14.047616 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:13.823053 24838 leader_election.cc:288] T dfa25b5175f14002a8b5ec8869d7c1d7 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:13.830245 24971 raft_consensus.cc:3037] T b730f70f25e440e68595cc248b40ee98 P 93578836968649f6bfcaecc709871412 [term 3 FOLLOWER]: Advancing to term 4
I20260812 19:14:14.044955 25325 raft_consensus.cc:482] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:13.855086 25315 raft_consensus.cc:2726] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260812 19:14:14.179258 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 59629.2us
I20260812 19:14:13.830193 25183 raft_consensus.cc:482] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 63db10063a7d4eb2a9cc9dc952654706)
I20260812 19:14:14.069345 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1f5b1c2f184b59b8f452b7f8351692" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:14.064239 21150 raft_consensus.cc:2379] T a4c8327f92104d69b0af3f6e7bf28d17 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 9: Already voted for candidate 93578836968649f6bfcaecc709871412 in this term.
I20260812 19:14:14.063359 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:14.181227 22045 raft_consensus.cc:2454] T 4580f6a22878438bb77f8d43fbddf459 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 5.
I20260812 19:14:14.125938 21329 rpcz_store.cc:276] 0812 19:14:12.793910 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:13.045858 (+251948us) service_pool.cc:227] Handling call
0812 19:14:13.046248 (+ 390us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:13.046546 (+ 298us) raft_consensus.cc:1520] Early marking committed up to index 5
0812 19:14:13.075128 (+ 28582us) log.cc:842] Serialized 14 byte log entry
0812 19:14:14.124612 (+1049484us) spinlock_profiling.cc:243] Waited 7.58 ms on lock 0x7b1800237d60. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52dfa1d8b1 00007f52dfa2a661 00007f52d973582c 00007f52d6b6c9dc 00007f52df997a39 00007f52d6b898bb 00007f52d6b87cbf 00007f52d6b87155 00007f52d9a9e3d0 00007f52d9ab0576 00007f52d9ac10d7
0812 19:14:14.124769 (+ 157us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:14.124841 (+ 72us) raft_consensus.cc:1631] Marking committed up to 5
0812 19:14:14.124920 (+ 79us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:14.125022 (+ 102us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:14.125209 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":28197,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":57,"raft.queue_time_us":106509,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":26997,"spinlock_wait_cycles":17444736,"thread_start_us":144627,"threads_started":1}
I20260812 19:14:14.062669 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580f6a22878438bb77f8d43fbddf459" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:14.051259 25303 consensus_meta.cc:287] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.422s user 0.013s sys 0.010s
I20260812 19:14:14.091127 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04322c3cc8b46a190cd46147e1c66d9" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:14.184949 21156 raft_consensus.cc:2454] T b04322c3cc8b46a190cd46147e1c66d9 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 8.
W20260812 19:14:14.056037 25306 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.976s user 0.018s sys 0.037s
I20260812 19:14:14.073671 21155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:14.038708 25044 raft_consensus.cc:2781] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader election won for term 8
I20260812 19:14:14.045009 25023 raft_consensus.cc:2781] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader election won for term 9
I20260812 19:14:14.098376 25184 consensus_queue.cc:260] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:14.117120 25390 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260812 19:14:14.151793 21327 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.003s user 0.009s sys 0.057s
W20260812 19:14:14.196897 21327 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.048s user 0.010s sys 0.058s
I20260812 19:14:14.128005 21321 rpcz_store.cc:276] 0812 19:14:13.008186 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:14.126601 (+1118415us) service_pool.cc:227] Handling call
0812 19:14:14.127027 (+ 426us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:14.127226 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:14.127257 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:14.127300 (+ 43us) raft_consensus.cc:1631] Marking committed up to 6
0812 19:14:14.127333 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:14.127386 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:14.127506 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:14.045397 25322 raft_consensus.cc:729] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:14.152760 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6521960ce6040af919a9e227d2211ec" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
W20260812 19:14:14.075157 25368 consensus_meta.cc:287] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 1.049s user 0.015s sys 0.000s
I20260812 19:14:14.052834 25330 raft_consensus.cc:482] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 93578836968649f6bfcaecc709871412)
I20260812 19:14:13.862113 25383 consensus_queue.cc:1046] T 85aa3ef35cc045a393fcbb13f15370a7 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.238s
I20260812 19:14:13.919467 25268 consensus_queue.cc:1035] T 0e242bf1e9fb45a197a8eaa86bf09f44 P f741102bfe1441d1b453e5a838649059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.213s
W20260812 19:14:14.151525 25301 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.905s user 0.007s sys 0.075s
I20260812 19:14:13.809674 25374 raft_consensus.cc:504] T d58350d75f2f4a61b79148bd023ff99c P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:14.058812 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c3d64ee3474fd0b1c7fd42a6cd30d3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:14.048568 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31c6a2accba4d29ba096500b9d1a911" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
I20260812 19:14:14.095227 24996 raft_consensus.cc:482] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d1e1da0b9585460e8c365bf55679d4fa)
W20260812 19:14:14.202386 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 166156us
I20260812 19:14:14.214087 21152 raft_consensus.cc:2379] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1e1da0b9585460e8c365bf55679d4fa in current term 8: Already voted for candidate f741102bfe1441d1b453e5a838649059 in this term.
I20260812 19:14:14.216540 21203 leader_election.cc:302] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:14.217614 22056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa25b5175f14002a8b5ec8869d7c1d7" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:14.218456 21324 raft_consensus.cc:1226] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [term 7 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []
I20260812 19:14:14.058091 25260 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:14.058940 25215 raft_consensus.cc:504] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:14.221942 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fc49d178354e629e97311498b381ec" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:14.221918 21200 leader_election.cc:302] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:14.228260 21321 raft_consensus.cc:1226] T 392684a63faf471c902391da6a7b0308 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 5. Current term is 6. Ops: []
W20260812 19:14:14.228739 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 195894us
W20260812 19:14:14.235057 21236 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1701) took 1426 ms (client timeout 1793 ms). Trace:
I20260812 19:14:14.240438 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "d1e1da0b9585460e8c365bf55679d4fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "93578836968649f6bfcaecc709871412" is_pre_election: true
W20260812 19:14:14.240844 21236 rpcz_store.cc:269] 0812 19:14:12.809070 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:14.058054 (+1248984us) service_pool.cc:227] Handling call
0812 19:14:14.234967 (+176913us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:14.241262 21324 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P 93578836968649f6bfcaecc709871412 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 6.
I20260812 19:14:14.241825 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32332933db0a4070bbf6fbabe17e716d" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
W20260812 19:14:14.242676 21236 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1703) took 1381 ms (client timeout 1817 ms). Trace:
W20260812 19:14:14.243095 21236 rpcz_store.cc:269] 0812 19:14:12.861667 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:14.241529 (+1379862us) service_pool.cc:227] Handling call
0812 19:14:14.242576 (+ 1047us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:14.243651 21200 leader_election.cc:302] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
I20260812 19:14:14.245620 22056 raft_consensus.cc:2379] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 in current term 5: Already voted for candidate 63db10063a7d4eb2a9cc9dc952654706 in this term.
I20260812 19:14:14.240458 21287 leader_election.cc:302] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:14.199092 25323 raft_consensus.cc:482] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:14.250239 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c656db261bd45f8b421339ba53dd87f" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
W20260812 19:14:14.228993 25301 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.983s user 0.007s sys 0.076s
I20260812 19:14:14.227979 25184 raft_consensus.cc:3037] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:14.211621 25138 consensus_queue.cc:1046] T 3dcaba8d2e2c42b48d7fa20deee5b73f P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260812 19:14:14.280937 21151 raft_consensus.cc:2454] T 6f1f5b1c2f184b59b8f452b7f8351692 P f741102bfe1441d1b453e5a838649059 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e1da0b9585460e8c365bf55679d4fa in term 10.
W20260812 19:14:14.278870 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 62428us
I20260812 19:14:14.211205 25390 raft_consensus.cc:3037] T aff4931ad1bb4657aa3650a15b43ddcd P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:14.282370 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f11f84389864fddb8350b0f94829938" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:14.284375 25004 raft_consensus.cc:2726] T 4c656db261bd45f8b421339ba53dd87f P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260812 19:14:14.231905 25368 raft_consensus.cc:2726] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260812 19:14:14.224684 25023 raft_consensus.cc:686] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [term 9 LEADER]: Becoming Leader. State: Replica: 93578836968649f6bfcaecc709871412, State: Running, Role: LEADER
W20260812 19:14:14.373588 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 130050us
I20260812 19:14:14.287978 25246 raft_consensus.cc:2781] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 19:14:14.295641 25359 raft_consensus.cc:3032] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Stepping down as leader of term 5
I20260812 19:14:14.208930 25191 consensus_queue.cc:1046] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.082s
I20260812 19:14:14.189806 25325 raft_consensus.cc:504] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:14.212172 24838 raft_consensus.cc:2741] T 49361681c5c1421594d10cd52061ec44 P 93578836968649f6bfcaecc709871412 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
W20260812 19:14:14.227418 25303 consensus_queue.cc:1159] T f2bb026b4ce64ee6beb14a67a87b06af P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f741102bfe1441d1b453e5a838649059" responder_term: 7 status { last_received { term: 6 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260812 19:14:14.236030 25330 raft_consensus.cc:504] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:14.361045 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 120654us
I20260812 19:14:14.196347 25183 raft_consensus.cc:504] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:14.365497 22008 leader_election.cc:334] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 93578836968649f6bfcaecc709871412 (127.19.70.131:33285): Timed out: RequestConsensusVote RPC to 127.19.70.131:33285 timed out after 1.620s (SENT)
I20260812 19:14:14.224993 24606 raft_consensus.cc:482] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:14.370656 21236 raft_consensus.cc:2454] T 3f11f84389864fddb8350b0f94829938 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 6.
W20260812 19:14:14.414999 21236 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1694) took 1645 ms (client timeout 1926 ms). Trace:
W20260812 19:14:14.415548 21236 rpcz_store.cc:269] 0812 19:14:12.769107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:14.282035 (+1512928us) service_pool.cc:227] Handling call
0812 19:14:14.414902 (+132867us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:14.374754 25383 raft_consensus.cc:482] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:14.224212 25164 raft_consensus.cc:482] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
I20260812 19:14:14.283146 25389 raft_consensus.cc:2726] T eadd5203f7ca4c04b02bd28bea6b591c P 93578836968649f6bfcaecc709871412 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 19:14:14.287329 25237 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.014s user 0.019s sys 0.019s
I20260812 19:14:14.375180 21200 leader_election.cc:302] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa; no voters: f741102bfe1441d1b453e5a838649059
W20260812 19:14:14.378387 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 204235us
W20260812 19:14:14.379031 21204 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.015s sys 0.011s
I20260812 19:14:14.374498 25394 raft_consensus.cc:2781] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 19:14:14.381569 25357 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807)
I20260812 19:14:14.402539 24843 raft_consensus.cc:2781] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader election won for term 9
I20260812 19:14:14.386668 25359 raft_consensus.cc:729] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
W20260812 19:14:14.384861 21288 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.132:39579, user_credentials={real_user=slave}} blocked reactor thread for 98367.6us
I20260812 19:14:14.407961 22008 leader_election.cc:302] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
I20260812 19:14:14.368968 25306 raft_consensus.cc:482] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f741102bfe1441d1b453e5a838649059)
W20260812 19:14:14.433382 25237 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.167s user 0.021s sys 0.022s
I20260812 19:14:14.408030 21287 leader_election.cc:302] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:14.417026 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59dcce5cc3a24cc088f6eb1fe3b3a021" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:14.282956 25312 raft_consensus.cc:482] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:14.377288 25246 raft_consensus.cc:482] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:14.375301 25384 leader_election.cc:288] T 932fac0d61e94da3a0f9c11ae2f597c3 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:14.483945 25405 raft_consensus.cc:2726] T fc50dccc6345495aaec42b51dc30d299 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260812 19:14:14.434092 25164 raft_consensus.cc:504] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:14.485371 25246 raft_consensus.cc:3037] T 18c9ea4c9e414d1093d28b6c91e19ff0 P f741102bfe1441d1b453e5a838649059 [term 6 FOLLOWER]: Advancing to term 7
I20260812 19:14:14.486665 25312 raft_consensus.cc:504] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:14.488418 25164 leader_election.cc:288] T aefbee13b3b74848be07bfcab278a3c8 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:14.488849 21244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1669) took 2481 ms (client timeout 1988 ms). Trace:
W20260812 19:14:14.489313 21244 rpcz_store.cc:269] 0812 19:14:12.006979 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:12.751145 (+744166us) service_pool.cc:227] Handling call
0812 19:14:14.488626 (+1737481us) spinlock_profiling.cc:243] Waited 502 ms on lock 0x7b54000b5754. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:14.488761 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1155311872}
I20260812 19:14:14.431416 21150 raft_consensus.cc:1226] T d6521960ce6040af919a9e227d2211ec P f741102bfe1441d1b453e5a838649059 [term 8 FOLLOWER]: Rejecting Update request from peer 63db10063a7d4eb2a9cc9dc952654706 for earlier term 7. Current term is 8. Ops: []
W20260812 19:14:14.430284 21200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 55223.4us
W20260812 19:14:14.430857 21204 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.578s user 0.015s sys 0.012s
I20260812 19:14:14.375684 21203 leader_election.cc:302] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412, f741102bfe1441d1b453e5a838649059
W20260812 19:14:14.432569 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 188078us
W20260812 19:14:14.428366 22047 consensus_meta.cc:287] T c31c6a2accba4d29ba096500b9d1a911 P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.615s user 0.012s sys 0.000s
I20260812 19:14:14.493108 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4e2cce3884f938f1efd2cbbe1a4a3" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:14.397411 25268 raft_consensus.cc:482] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:14.385545 21329 raft_consensus.cc:1226] T 3dcaba8d2e2c42b48d7fa20deee5b73f P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Rejecting Update request from peer d1e1da0b9585460e8c365bf55679d4fa for earlier term 8. Current term is 9. Ops: []
I20260812 19:14:14.429773 24606 raft_consensus.cc:504] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:14.439064 25023 consensus_queue.cc:237] T a4c8327f92104d69b0af3f6e7bf28d17 P 93578836968649f6bfcaecc709871412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
W20260812 19:14:14.441448 21157 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51004 (request call id 480) took 1823 ms (client timeout 2239 ms). Trace:
W20260812 19:14:14.501471 21157 rpcz_store.cc:269] 0812 19:14:12.618518 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:12.618741 (+ 223us) service_pool.cc:227] Handling call
0812 19:14:14.441200 (+1822459us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b54002931d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aaa2ae 00007f52de9e7c27 00007f52d80dbc04 00007f52d80dbb92 00007f52d80dbac1 00007f52d80dba3c 00007f52d80dacf2 00007f52d7edc94c 00007f52d7edbd86 00007f52d7edb72f
0812 19:14:14.441373 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1113526656}
I20260812 19:14:14.395785 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e03c303e40432e8f5da639f7edff82" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:14.443630 21235 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37894 (request call id 1656) took 5137 ms. Trace:
W20260812 19:14:14.451409 25392 log.cc:924] Time spent T a8161580b52441ab9a17a0e1597f243b P 93578836968649f6bfcaecc709871412: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
W20260812 19:14:14.453874 25391 log.cc:924] Time spent T 2f5090bb13ea4aa792b27b04a7a79688 P 63db10063a7d4eb2a9cc9dc952654706: Append to log took a long time: real 0.155s user 0.001s sys 0.000s
I20260812 19:14:14.455506 25183 leader_election.cc:288] T fc50dccc6345495aaec42b51dc30d299 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 93578836968649f6bfcaecc709871412 (127.19.70.131:33285), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:14.455798 25322 consensus_queue.cc:260] T 38dbcfe527fe4701863d72b9bb480ccf P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:14.428332 25399 raft_consensus.cc:482] T f2bb026b4ce64ee6beb14a67a87b06af P f741102bfe1441d1b453e5a838649059 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:14.456730 25215 leader_election.cc:288] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:14.476981 25306 raft_consensus.cc:504] T 41fc49d178354e629e97311498b381ec P 63db10063a7d4eb2a9cc9dc952654706 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:14.458199 25330 leader_election.cc:288] T ad64c74a7de6449c8152bc0e093a59db P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:14.458861 25325 leader_election.cc:288] T 66cbc2eec68f4a328fa4cf6c0de63656 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:14.459080 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a070b29d02d24ef4b36656fcac6d68d9" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f741102bfe1441d1b453e5a838649059" is_pre_election: true
I20260812 19:14:14.460397 25260 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:14.557163 21154 raft_consensus.cc:2454] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63db10063a7d4eb2a9cc9dc952654706 in term 9.
I20260812 19:14:14.249070 24996 raft_consensus.cc:504] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:14.463824 22008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 98367.5us
I20260812 19:14:14.468627 21236 raft_consensus.cc:3037] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:14.461162 25359 consensus_queue.cc:260] T 392684a63faf471c902391da6a7b0308 P d1e1da0b9585460e8c365bf55679d4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:14.470232 24843 raft_consensus.cc:686] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [term 9 LEADER]: Becoming Leader. State: Replica: d1e1da0b9585460e8c365bf55679d4fa, State: Running, Role: LEADER
I20260812 19:14:14.437160 25394 raft_consensus.cc:482] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:14.471514 25393 raft_consensus.cc:2726] T 784d71ae37454b3a9e8023f268428468 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260812 19:14:14.472952 25238 consensus_meta.cc:287] T 66cbc2eec68f4a328fa4cf6c0de63656 P f741102bfe1441d1b453e5a838649059: Time spent flushing consensus metadata: real 1.448s user 0.011s sys 0.005s
W20260812 19:14:14.448060 25362 consensus_meta.cc:287] T 18c9ea4c9e414d1093d28b6c91e19ff0 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.183s user 0.017s sys 0.000s
W20260812 19:14:14.457825 25320 consensus_meta.cc:287] T 6f1f5b1c2f184b59b8f452b7f8351692 P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.202s user 0.013s sys 0.000s
W20260812 19:14:14.482774 21287 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 74823us
I20260812 19:14:14.593219 21287 leader_election.cc:302] T 2117fe1b5b8c42758a313da21e01a17c P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:14.483284 22055 raft_consensus.cc:1226] T dfa25b5175f14002a8b5ec8869d7c1d7 P 63db10063a7d4eb2a9cc9dc952654706 [term 5 FOLLOWER]: Rejecting Update request from peer f741102bfe1441d1b453e5a838649059 for earlier term 4. Current term is 5. Ops: []
I20260812 19:14:14.494180 21244 raft_consensus.cc:2454] T 0eb4e2cce3884f938f1efd2cbbe1a4a3 P d1e1da0b9585460e8c365bf55679d4fa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 5.
W20260812 19:14:14.486469 21114 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.130:33807, user_credentials={real_user=slave}} blocked reactor thread for 2.01004e+06us
W20260812 19:14:14.595117 21244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1702) took 1743 ms (client timeout 1865 ms). Trace:
W20260812 19:14:14.595484 21244 rpcz_store.cc:269] 0812 19:14:12.851999 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:14.490275 (+1638276us) service_pool.cc:227] Handling call
0812 19:14:14.595054 (+104779us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:14.509450 25268 raft_consensus.cc:504] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:14.597401 21204 leader_election.cc:302] T 4580f6a22878438bb77f8d43fbddf459 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, d1e1da0b9585460e8c365bf55679d4fa; no voters: 93578836968649f6bfcaecc709871412
I20260812 19:14:14.447017 25138 raft_consensus.cc:2781] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 19:14:14.599751 25138 raft_consensus.cc:482] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 19:14:14.600244 25138 raft_consensus.cc:3037] T 1c678904ece144c29ee1c70ff4568969 P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Advancing to term 8
I20260812 19:14:14.503516 21235 rpcz_store.cc:276] 0812 19:14:09.306688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:11.504750 (+2198062us) service_pool.cc:227] Handling call
0812 19:14:11.516027 (+ 11277us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:11.641869 (+125842us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:14.443294 (+2801425us) spinlock_profiling.cc:243] Waited 75.2 ms on lock 0x7b54002c47d4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:14.443550 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":172916992}
I20260812 19:14:14.522372 21112 leader_election.cc:302] T 392684a63faf471c902391da6a7b0308 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 93578836968649f6bfcaecc709871412, d1e1da0b9585460e8c365bf55679d4fa
I20260812 19:14:14.466673 25044 raft_consensus.cc:686] T 49361681c5c1421594d10cd52061ec44 P f741102bfe1441d1b453e5a838649059 [term 8 LEADER]: Becoming Leader. State: Replica: f741102bfe1441d1b453e5a838649059, State: Running, Role: LEADER
W20260812 19:14:14.457180 25193 consensus_meta.cc:287] T a8161580b52441ab9a17a0e1597f243b P d1e1da0b9585460e8c365bf55679d4fa: Time spent flushing consensus metadata: real 1.541s user 0.017s sys 0.000s
I20260812 19:14:14.489313 25312 leader_election.cc:288] T 0e242bf1e9fb45a197a8eaa86bf09f44 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:14.462275 25301 consensus_queue.cc:1046] T 7825e8ab0b2e4d7abbb7dd19c9e3ad69 P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.072s
W20260812 19:14:14.569943 21203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 194368us
I20260812 19:14:14.570307 25413 raft_consensus.cc:2726] T a4c8327f92104d69b0af3f6e7bf28d17 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260812 19:14:14.613251 21112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.70.131:33285, user_credentials={real_user=slave}} blocked reactor thread for 90994.3us
I20260812 19:14:14.513957 25357 raft_consensus.cc:482] T 8a915c8829aa4e1fae0110e009af0416 P 93578836968649f6bfcaecc709871412 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:14.516898 25133 leader_election.cc:288] T 2117fe1b5b8c42758a313da21e01a17c P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
I20260812 19:14:14.541796 25183 raft_consensus.cc:482] T 2f5090bb13ea4aa792b27b04a7a79688 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:14.586267 21327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50042 (request call id 467) took 1830 ms. Trace:
I20260812 19:14:14.413527 25323 raft_consensus.cc:504] T c6309c277c104b23a58d4824d764d397 P 63db10063a7d4eb2a9cc9dc952654706 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
W20260812 19:14:14.588770 22011 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.70.129:36551, user_credentials={real_user=slave}} blocked reactor thread for 64535us
I20260812 19:14:14.435096 25383 raft_consensus.cc:504] T aff4931ad1bb4657aa3650a15b43ddcd P d1e1da0b9585460e8c365bf55679d4fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "63db10063a7d4eb2a9cc9dc952654706" member_type: VOTER last_known_addr { host: "127.19.70.132" port: 39579 } attrs { promote: false } }
I20260812 19:14:14.594425 21287 leader_election.cc:302] T f2bb026b4ce64ee6beb14a67a87b06af P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93578836968649f6bfcaecc709871412; no voters: d1e1da0b9585460e8c365bf55679d4fa, f741102bfe1441d1b453e5a838649059
I20260812 19:14:14.619858 21327 rpcz_store.cc:276] 0812 19:14:12.756182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:12.964203 (+208021us) service_pool.cc:227] Handling call
0812 19:14:12.964608 (+ 405us) raft_consensus.cc:1457] Updating replica for 1 ops
0812 19:14:13.099608 (+135000us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:13.099644 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0812 19:14:13.103887 (+ 4243us) log.cc:842] Serialized 26 byte log entry
0812 19:14:14.229142 (+1125255us) raft_consensus.cc:1631] Marking committed up to 8
0812 19:14:14.229211 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:14.229299 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0812 19:14:14.585972 (+356673us) raft_consensus.cc:1671] finished
0812 19:14:14.585996 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:14.586188 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6121,"raft.queue_time_us":160051,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":23234,"thread_start_us":1158785,"threads_started":2,"wal-append.queue_time_us":1081767}
I20260812 19:14:14.599996 25268 leader_election.cc:288] T 3dcaba8d2e2c42b48d7fa20deee5b73f P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 93578836968649f6bfcaecc709871412 (127.19.70.131:33285)
W20260812 19:14:14.498061 25265 consensus_meta.cc:287] T 9ce6040ab9fc41539e7fc09b3f9d05dd P 63db10063a7d4eb2a9cc9dc952654706: Time spent flushing consensus metadata: real 0.696s user 0.008s sys 0.005s
I20260812 19:14:14.541393 25331 consensus_queue.cc:1046] T d6521960ce6040af919a9e227d2211ec P 63db10063a7d4eb2a9cc9dc952654706 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
W20260812 19:14:14.605162 25338 consensus_meta.cc:287] T 4580f6a22878438bb77f8d43fbddf459 P 93578836968649f6bfcaecc709871412: Time spent flushing consensus metadata: real 1.561s user 0.016s sys 0.000s
I20260812 19:14:14.592592 25384 raft_consensus.cc:482] T e7e49dcb1cac44ee91df33a2abea1dd0 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 19:14:14.605043 24843 consensus_queue.cc:237] T 7dfc261e3ffb48ddaa432f57bf7aad22 P d1e1da0b9585460e8c365bf55679d4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f741102bfe1441d1b453e5a838649059" member_type: VOTER last_known_addr { host: "127.19.70.129" port: 36551 } } peers { permanent_uuid: "d1e1da0b9585460e8c365bf55679d4fa" member_type: VOTER last_known_addr { host: "127.19.70.130" port: 33807 } } peers { permanent_uuid: "93578836968649f6bfcaecc709871412" member_type: VOTER last_known_addr { host: "127.19.70.131" port: 33285 } }
I20260812 19:14:14.607497 24996 leader_election.cc:288] T e6eaa29a6fe94976a6124729431bd8c8 P 93578836968649f6bfcaecc709871412 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d1e1da0b9585460e8c365bf55679d4fa (127.19.70.130:33807), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:14.607903 24606 leader_election.cc:288] T dc64ef202b904bcf809760ddfcd1f140 P d1e1da0b9585460e8c365bf55679d4fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f741102bfe1441d1b453e5a838649059 (127.19.70.129:36551), 63db10063a7d4eb2a9cc9dc952654706 (127.19.70.132:39579)
I20260812 19:14:14.612170 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eaa29a6fe94976a6124729431bd8c8" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:14.623287 21235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e49dcb1cac44ee91df33a2abea1dd0" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"
I20260812 19:14:14.626013 21319 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51568 (request call id 1975) took 1837 ms. Trace:
I20260812 19:14:14.614115 21112 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P f741102bfe1441d1b453e5a838649059 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f741102bfe1441d1b453e5a838649059; no voters: 63db10063a7d4eb2a9cc9dc952654706, 93578836968649f6bfcaecc709871412
I20260812 19:14:14.633288 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfc261e3ffb48ddaa432f57bf7aad22" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa" is_pre_election: true
I20260812 19:14:14.618129 21236 raft_consensus.cc:2454] T 59dcce5cc3a24cc088f6eb1fe3b3a021 P d1e1da0b9585460e8c365bf55679d4fa [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93578836968649f6bfcaecc709871412 in term 9.
I20260812 19:14:14.635068 21319 rpcz_store.cc:276] 0812 19:14:12.788296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:13.021690 (+233394us) service_pool.cc:227] Handling call
0812 19:14:13.022072 (+ 382us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 19:14:13.233167 (+211095us) raft_consensus.cc:1520] Early marking committed up to index 0
0812 19:14:13.233232 (+ 65us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0812 19:14:13.233293 (+ 61us) raft_consensus.cc:1631] Marking committed up to 7
0812 19:14:13.233341 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 19:14:14.625623 (+1392282us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b5400295ed4. stack: 00007f52d6b58033 00007f52d6b57f30 00007f52d64b6534 00007f52e01b68b4 00007f52e01b6849 00007f52d9a2b471 00007f52d9aa6f51 00007f52d9aa599e 00007f52de9e72e9 00007f52d80d9684 00007f52d80d9612 00007f52d80d9541 00007f52d80d94bc 00007f52d80d8772 00007f52d7edc94c 00007f52d7edbd86
0812 19:14:14.625699 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0812 19:14:14.625920 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":484830592}
I20260812 19:14:14.621560 22011 leader_election.cc:302] T a070b29d02d24ef4b36656fcac6d68d9 P 63db10063a7d4eb2a9cc9dc952654706 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63db10063a7d4eb2a9cc9dc952654706, f741102bfe1441d1b453e5a838649059; no voters:
I20260812 19:14:14.586637 25394 raft_consensus.cc:3037] T b04322c3cc8b46a190cd46147e1c66d9 P d1e1da0b9585460e8c365bf55679d4fa [term 8 FOLLOWER]: Advancing to term 9
I20260812 19:14:14.639688 22055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58350d75f2f4a61b79148bd023ff99c" candidate_uuid: "93578836968649f6bfcaecc709871412" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "63db10063a7d4eb2a9cc9dc952654706" is_pre_election: true
I20260812 19:14:14.644924 22055 raft_consensus.cc:2396] T d58350d75f2f4a61b79148bd023ff99c P 63db10063a7d4eb2a9cc9dc952654706 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93578836968649f6bfcaecc709871412 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260812 19:14:14.640393 21244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1695) took 1871 ms (client timeout 1989 ms). Trace:
W20260812 19:14:14.645594 21244 rpcz_store.cc:269] 0812 19:14:12.769275 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:14.610477 (+1841202us) service_pool.cc:227] Handling call
0812 19:14:14.640326 (+ 29849us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 19:14:14.640039 21236 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37894 (request call id 1705) took 1688 ms (client timeout 1752 ms). Trace:
W20260812 19:14:14.647460 21236 rpcz_store.cc:269] 0812 19:14:12.951660 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 19:14:14.416588 (+1464928us) service_pool.cc:227] Handling call
0812 19:14:14.639975 (+223387us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 19:14:14.648824 21236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65db14a4838d40218947e8bd3420e892" candidate_uuid: "63db10063a7d4eb2a9cc9dc952654706" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d1e1da0b9585460e8c365bf55679d4fa"